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1 Artificial intelligence and 
international conflict in 
cyberspace 


Exploring three sets of issues 


Fabio Cristiano, Dennis Broeders, Francois Delerue, 
Frederick Douzet and Aude Gery 


Introduction 


Over the last three decades, cyberspace developed into a crucial frontier and 
issue of international conflict. Disproving the initial fear-mongering expec- 
tations of fully-fledged wars occurring in and through cyberspace, this con- 
flict increasingly unfolds ‘away from” the traditional categories and thresholds 
of war and peace.! As argued by Lucas Kello, cyberspace is neither truly at 
war nor at peace but maintains a constant condition of ‘unpeace.”” Interna- 
tional conflict in cyberspace primarily occurs in the so-called grey zone and 
often pertains to the domain of information, data, and their manipulation, 
culminating in acts of espionage, sabotage, and subversion. As empirical evi- 
dence overwhelmingly shows, confrontation in cyberspace mostly consists of 
low-impact hacking, espionage, disinformation, and surveillance. In light of 
this, recent scholarly work interrogates whether we should consider conflict 
in cyberspace as an ‘intelligence competition’ rather than through the lenses 
of traditional warfare. At the same time, this does not mean that we should 
think of cyberspace as the peaceful, yet ungoverned and ungovernable, oasis 
envisioned by cyber libertarians in the early days of the internet? — quite the 
opposite. States now conventionally conceive of cyberspace as an issue of na- 
tional security and increasingly safeguard and promote their national interests 
through both defensive strategies and offensive operations in cyberspace.° 

In a context where data and information have become increasingly impor- 
tant, it comes as no surprise that the development and application of artificial 
intelligence (AI) have gained momentum in the various discourses about in- 
ternational conflict in cyberspace. Al technologies — such as machine learn- 
ing, natural language processing, quantum computing, neural networks, and 
deep learning — provide military and intelligence agencies with new opera- 
tional solutions for predicting and countering threats as well as for conduct- 
ing offensive operations in cyberspace. Besides automating the production 
of knowledge about cyber threats, AI can also automate decision-making, 
which could ‘dilute’ the role of (human) political agency as an element of 
international conflict in cyberspace. Concerns at the core of the international 
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debate about Lethal Autonomous Weapon Systems (LAWS) would then also 
enter the debate about cyber conflict. Moreover, the operational entangle- 
ment of AI technologies in cyberspace further blurs the already contested 
lines between defence and offence in cyberspace,’ while also challenging the 
divide between cyber conflict and information operations.“ Besides opening 
up new operational milieus, the adoption of Al-enhanced cyber capabilities 
also represents an important strategic asset for states, with the ongoing global 
race towards the adoption of these technologies fully embedded in broader 
geopolitical conflicts, deterrence, securitisation strategies, and techno- 
nationalist narratives, such as those about digital sovereignty.” 

The entanglement of AI technologies with cyber conflict raises several is- 
sues primarily related to human-machine interaction, the role of (big) data 
in society, great powers competition, and regulation. While creating the ‘il- 
lusion’ of scientific and data-driven security, delegating security functions to 
independent machines might expose networks to a whole variety of new risks 
emerging because of autonomy and automation. Potential biases in the me- 
chanical processing of data can lead to miscalculations and the creation of a 
broader ‘attack surface’ and vulnerability for the systems that AI purports to 
protect. Similarly, the global race towards the acquisition of these technologies 
also risks further intensifying and polarising international conflict in cyber- 
space.!! For these reasons, AI technologies have also gained interest as a nor- 
mative issue across ethical and legal debates on responsible (state) behaviour in 
cyberspace — although the debate about autonomy has not fully crossed over 
from the military domain ‘proper’ to that of cyber conflict yet.!? As this vol- 
ume shows, specific regulatory frameworks and legislations might be required 
to capture AI as both a potential asset and threat to national security and to the 
‘open and secure’ cyberspace that some countries seek to uphold. 

With the intent of exploring the question ‘what is at stake with the use of 
automation in international conflict in cyberspace through AI?’, this volume 
focuses on three themes, namely: (1) technical and operational, (2) strategic 
and geopolitical, and (3) normative and legal. These also constitute the three 
parts in which the chapters of this volume are organised. Scholarly work on the 
relationship between AI and conflict in cyberspace has been produced along 
somewhat rigid disciplinary boundaries and an even more rigid sociotechnical 
divide — wherein technical and social scholarship are seldomly brought into a 
conversation. This volume addresses these themes through a comprehensive 
and cross-disciplinary approach. In this sense, the organisation of the volume 
in three parts should not be considered as an analytical or, even less so, a disci- 
plinary demarcation. The remainder of this introductory chapter outlines, and 
provides context for, the main debates of each of the three parts of the volume. 


Technical and operational considerations 


AI has emerged as the defining technology of our times and seems to epit- 
omise the ultimate innovation that everybody wants and about which 
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everybody is ‘concerned.’ States often have a techno-optimistic view of new 
technologies and look favourably at the prospect of rationalising and perfect- 
ing governance through automation,!” with AI currently being applied to 
wide and diverse governance domains and issues. The allure of the concept of 
‘AL is perhaps best caught by the fact that many applications in government 
(and outside of it) would still be more aptly labelled as ‘classic’ automation 
rather than AI or the introduction of autonomy in systems. However, de- 
velopments in AI do start to permeate traditional governance by expanding 
the range, scale, and complexity of operations that can be meaningfully au- 
tomated, including those associated with cybersecurity. When compared to 
other governance branches, the application of AI technologies in cybersecu- 
rity represents however less of an innovation. Already in the 1990s, machine 
learning and neural networks were, for instance, applied to the filtering and 
classification of spam emails.!* After all, automation constitutes an inherent 
feature of internet technology and computation. What is relatively new, and 
of main interest for this volume, is the internationalisation and ‘datification’ 
of conflict in cyberspace, where the potential of Al marks a new operational 
phase through autonomy. 

From an operational perspective, Al technologies promise to contribute to 
one of the core dynamics of international conflict in cyberspace: the identifi- 
cation of vulnerabilities through timely and effective interpretation of data — 
for either defence or offence. That is, AI has the potential to make conflict 
in cyberspace more knowable and predictable. When considering aspects of 
automation and machine autonomy in the context of international conflict in 
cyberspace, the ability of intelligent machines to make operational choices — at 
different degrees of independence — points to the question of who the actual 
enactors of international conflict in cyberspace are. As will be further dis- 
cussed in the third part of this volume, this question is not only analytical or 
technical. Knowing who enacts conflict in cyberspace also intimately pertains 
to questions of responsibility! In a context where agency appears to be al- 
ready diluted through networks, and socio-technical assemblages, exploring 
the Al-cyber nexus primary means to explore human-mechanic interactions.'° 

The question of autonomy and AI raises an operational interrogative re- 
lated to the ‘place’ of humans in relation to the so-called ‘loop’ of oper- 
ational decision-making. This dilemma has been foremostly articulated in 
debates about LAWS where the central question remains whether humans 
shall be placed in, on, or outside of this loop.” In Chapter 2 of this vol- 
ume, Andrew Dwyer directly addresses this question by analysing the role of 
deep reinforcement learning (RL) algorithms to question assumptions that 
AI technologies make conflict in cyberspace more knowable. It argues that, 
by recognising, performing, and transforming the who, where, and how, of 
international conflict in cyberspace, AI constitutes more than an epistemic 
tool for improving operations. In this sense, the chapter also complicates nor- 
mative considerations about controllable and ethically accountable AI systems 
and about the place of the human ‘in’ the loop. 
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One of the core technical promises of AI for cybersecurity consists in what 
Tim Stevens defines as a shift “from known threats to the prolepsis of as-yet- 
unknown threats and into an anticipatory posture that has received much 
attention in the critical security literature.’!® In Chapter 3, Wesley Moy and 
Kacper Gradon explore the various potential applications of AI in the prop- 
agation of disinformation and misinformation, as well as in the context of 
hybrid and asymmetric warfare. By analysing two methodologies — namely 
“Generative Adversarial Networks’ and “Large Language Models’ — this chap- 
ter explains the relevance of AI for understanding how links are formed, how 
information is disseminated, and how information can influence opinions 
and actions in social networks. Taken together, the contributions to the first 
part of the volume indicate that, while enhancing operational efficiency, Al 
applications do not necessarily ‘make’ international conflict more known/ 
predictable and cybersecurity more human-centric. Rather, autonomy and 
automation further contribute to the problematic understanding of cyber- 
space as a primarily technical and operational issue or domain. 


Strategic and geopolitical considerations 


Looking beyond its technical possibilities and operational dilemmas, AI is set 
to become a constitutional component of economic, political, and military 
power in the digital age. With the return of great-power competition and the 
constant contestation and confrontation between states in cyberspace, AI is 
undergoing a process of securitisation that transforms this dual technology, 
primarily developed for civilian uses, into a matter of national security and 
sovereignty.” As a result, AI has become fully part of the contested global 
‘digital arms race,’ raising major concerns about the broader risks associated 
with its use for offensive purposes.” This evolution is not surprising. It is in 
line with the broader securitisation of cyberspace over the past three decades, 
quickly, but not always correctly, associated with its militarisation in the dis- 
course of states.7! 

With the rise of increasingly sophisticated and targeted state-sponsored 
cyberattacks since the late 2000s, cyberspace emerged as an imperative of 
securitisation and a new warfighting domain that required the mobilisation 
of exceptional means.”” The representation of cyberspace as predominantly 
a threat to national security is not self-evident given the complex challenges 
in this domain, such as those posed by criminal organisations to individual 
interests that can equally hurt the security of end users and the security and 
stability of cyberspace itself.” Other characterisations that could have pre- 
vailed such as economic risk, criminal danger, or threats to individual user 
privacy have increasingly taken a back seat to national and international secu- 
rity concerns.2* In the words of internet governance scholar, Milton Mueller 
‘cybersecurity is eating internet governance’ and is pushing out alternative 
framings.~° The security frame has progressively extended to all the digi- 
tal technologies that could be weaponised in the context of digital warfare, 
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including AI, and drives international competition over digital technologies. 
This competition is both embodied and increasingly shaped by the fierce 
competition between the United States and China over the production, 
control, use and governance of digital technologies. Adam Segal argues that 
during the 1990s and 2000s the integration of the Chinese and American 
economies was perceived as mutually beneficial, both politically and eco- 
nomically, political decision-makers now consider that the risks outweigh the 
benefits.” And in both state discourses, the issue of security is at the heart 
of the rivalry. It should be noted that China has launched a massive plan to 
become the world leader in AI by 2030, with a 150-billion-dollar industry.7’ 
That is, the talent war is on. 

The leadership of a few countries in AI capabilities also reveals uncom- 
fortable strategic dependencies for many other countries. It has triggered a 
debate in the European Union about the risk associated with these depend- 
encies and the need for strategic autonomy to ensure digital sovereignty. S 
But advancing AI technology appears to be a limited policy option to address 
these issues. In Chapter 4, Simona Soare questions the role of AI to advance 
European strategic autonomy in the field of security and defence. She argues 
that the adoption of AI is a ‘distraction’ as it introduces additional layers of 
complexity in the European defence while not contributing significantly to 
Europe’s strategic autonomy. On the one hand, the integration of AI in the 
EU decision-making processes and the conduct of operations is challenging 
because of the EU’s internal functioning in the field of defence. On the other 
hand, the lack of industrial capabilities and the strategic dependencies towards 
other powers are real and likely difficult to overcome. In Chapter 5 Arun 
Mohan Sukumar similarly demonstrates that relying on AI can introduce 
risks and strategic dependencies, as shown in the case of emerging powers. 
The chapter examines the role of AI in the development of public services, 
through examples of the health sector in Brazil, India and Singapore. It shows 
how, while states are urged to enhance data transparency and to develop 
digital services for their population, they become exposed to new risks that 
could set back progress in the digitalisation of states’ mission-critical systems 
for years. That is, they face a trade-off between furthering digitalisation and 
accepting more security risks, an instance that speaks to the importance of 
thinking about the Al-cyber nexus not only in technical/operational terms 
but also considering broader strategic implications. 

Armed forces worldwide have also recognised the strategic relevance of the 
Al-cyber nexus and have similarly engaged in a profound digital transforma- 
tion of their operations.” On the one hand, this has considerably increased 
their reliance on digital technologies and data. On the other, it has created 
new risks and vulnerabilities. Soldiers evolve in a new digital environment 
that profoundly transforms the way they operate and creates new challenges 
that are sometimes hard to fully comprehend and govern. In this environ- 
ment, Al offers promising new capabilities to improve the quality of intelli- 
gence, situational awareness, the conditions of training, the ability to operate 
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remotely, the precision and autonomy of weapon systems and, most impor- 
tantly, the speed and scope of action. As result, the race for AI is thus also a 
race for military power and superiority and, again, raises strategic problems 
that are intimately related to operational ones. This representation resonates 
with a vision deeply ingrained in the US military culture that technology can 
provide military superiority. In Chapter 6, Jeppe Jacobsen and Tobias Liebe- 
trau argue that this vision goes back a long time before AI and has dominated 
US military discourse since the Second Offset Strategy of the 1970s. That is, 
AI represents an operational innovation more than a strategic one. 

While providing further evidence to a presumed return of great powers 
competition, the military superiority approach also feeds the fears inspired by 
the technology and is a driver for developing offense over defence, to main- 
tain superiority over the enemy. But Al-enabled cyber capabilities might also 
convey the idea of control that is difficult if not illusory in cyberspace, given 
the highly dynamic nature of this environment.” And it does not take into 
consideration the vulnerabilities and associated risks that AI technology also 
brings about. Indeed, with the digital transformation of societies and armed 
forces, the attack surface keeps increasing. And while AI can considerably 
improve defence, the emphasis placed on offense could be a source of risk. 
Jeppe Jacobsen and Tobias Liebetrau demonstrate that the cyber arms race is 
not just a competition between great powers for Al-enabled cyber capabilities 
but also a specific arms race between offensive and defensive cyber capabil- 
ities, powered by AI. Given the lessons from discussions on how militaries 
balance offense and defence in cyberspace, they conclude that Al-enhanced 
cyber offensive capabilities are likely to dominate. And yet AI can backfire 
in many ways. As our societies grow increasingly dependent on digital tech- 
nologies, the securitisation of AI technology could have important spill-over 
effects on the overall level of cyber (in)stability. The ongoing race for data 
and its exploitation for strategic advantages further blurs the lines between 
military and civilian operations, with inextricable consequences for the pri- 
vate sector and civil society, raising new legal and normative challenges. 


Normative and legal considerations 


Stemming directly from the above-mentioned technical/operational and 
strategic/geopolitical considerations is the necessity of regulating the adop- 
tion and use of AI technologies in cyberspace. The development of cyber 
capabilities, on the one hand, and AI and its possible applications in cyber 
conflicts on the other, have posed a dilemma to states and other actors: they 
are interested in these new technologies — notably to enhance their own 
operational capabilities and strategic posture — but they are at the same time 
concerned about the potential consequences of these developments for in- 
ternational peace and security. This dilemma lies at the core of the third 
final part of this volume, which deals with the normative and legal questions 
raised by AI applications in cyberspace. To understand these, this section also 
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introduces the international processes in which these normative and legal dis- 
cussions are embedded and become deeply intertwined with states’ strategic 
considerations. 

On international cybersecurity, the United Nations General Assembly 
adopted its first resolution on “Developments in the field of information and tele- 
communications in the context of international security” in December 1998. Since 
2004, the United Nations General Assembly has established six successive 
Groups of Governmental Experts (GGE) on this topic. The first and the fifth 
GGE failed to adopt a consensus report, reportedly because of disagreement 
in the discussions on specific branches of international law. The impossibility 
of the fifth GGE to adopt a consensus report in June 2017 led to disagree- 
ment on how to proceed. In 2018, this resulted in the adoption of two con- 
current resolutions and the creation of two parallel processes, with largely 
the same mandate. In addition to the sixth GGE, an Open-Ended Working 
Group (OEWG) was established. In 2020, a new OEWG was established 
which will last until 2025 while there is as of, yet no new GGE planned.*! 
Moreover, since 2020, some States are advocating for a new process on this 
topic, a Program of Action (PoA) for advancing responsible state behaviour 
in cyberspace,*” which was welcomed in principle in November 2022 by 
the UN General Assembly. The second, third, fourth and sixth GGE as well 
as the first OEWG were successful in adopting consensus reports.°° These 
reports notably affirmed that international law is applicable to cyberspace 
and listed specific rules and principles of international law deemed particu- 
larly relevant in this context. They also listed 11 norms of responsible behav- 
iour in cyberspace. Taken together these reports constitute a framework of 
responsible State behaviour in cyberspace, encompassing international law 
and non-binding norms but also capacity building and confidence-building 
measures. Interestingly in the context of this book, the development of AI 
applications has never been mentioned in the GGE or OEWG reports, de- 
spite it being discussed in the 2019—2021 rounds of negotiation. While the 
issue did not make the cut of the 2021 consensus reports it does feature in the 
so-called Chair’s summary of the OEWG process in its section dedicated to 
‘Threats: “Pursuit of increasing automation and autonomy in ICT operations 
was put forward as a specific concern, as were actions that could lead to the 
reduction or disruption of connectivity, unintended escalation or effects that 
negatively impact third parties.” Moreover, both in the context of the UN 
negotiations and outside of it, states and other actors have started to voice 
concerns about the role of automation and autonomy in cyber operations.” 

The discussions on the international security dimensions of AI have been 
focusing on the development of LAWS. This matter was introduced in 2013 
in the agenda of the Meetings of High Contracting Parties to the Convention 
on Prohibitions or Restrictions on the Use of Certain Conventional Weapons 
(CCW). After a few informal meetings, these discussions took a similar path 
as the ones on international cybersecurity, with the establishment a GGE 
in 2016 which adopted 11 guiding principles on LAWS in 2019.36 Through 
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these principles, the GGE affirmed the applicability of international law and 
in particular international humanitarian law as well as a series of ethical and 
non-binding principles. Surprisingly, Cybersecurity is only briefly mentioned 
in the sixth principle as one of the “appropriate non-physical safeguards [that] 
should be considered [w]hen developing or acquiring new weapons systems 
based on emerging technologies in the area of lethal autonomous weapons 
systems.” There is, however, no mention of autonomous cyber capabilities. 
Even though the link between cyber security and AI has been made in the 
context of the OECD“ and in the UNESCO Recommendation on the Ethics 
of Artificial Intelligence,” both of these documents steer clear of national and 
international security. So until now, ‘cyber’ and ‘AI’ seem to be ships passing 
in the night in the UN’s first committee. 

This ‘absence’ is at the heart of the third part of this volume. To navigate 
this vacuum at the international level. Taddeo, McNeish, Blanchard, and Ed- 
gar discuss in Chapter 7 the efforts to define ethical frameworks to guide the 
use of Al in the defence domain at the domestic level — through the case of 
the United Kingdom — and propose a possible framework, articulated around 
five principles: justified and overridable uses; just and transparent systems 
and processes; human moral responsibility; meaningful human control; and, 
finally, reliable AI systems. At the core of these ethical considerations are the 
matter of technological autonomy and the need for some form of human con- 
trol, involvement, or override: again, where does the human fit in ‘the loop’? 
Going back to the international level, in Chapter 8 Louis Perez navigates the 
different discussion streams at the UN on Cyber on the one hand and LAWS 
on the other, before discussing how the current approach to LAWS could 
also be applied to autonomous cyber operations. Reflecting on the definition 
of LAWS, this chapter addresses the vital question of whether autonomous 
cyber capabilities could be considered LAWS and thus be concerned by the 
discussions on international law and ethics taking place in the framework of 
the CCW. In Chapter 9, Jack Kenny focuses on a specific principle of interna- 
tional law, the principle of non-intervention, that has been discussed exten- 
sively by the GGE and the OEWG. Building on these discussions, as well as 
on the existing scholarship on the application of this principle in cyberspace, 
this chapter looks at the specific challenges raised by automation for this 
principle with a specific focus on its coercion requirement. By going back to 
one of the operational dilemmas discussed earlier, the chapter elucidates this 
normative discussion through the analysis of different examples related to the 
interference in electoral processes using cyber means with a certain degree 
of autonomy. 

The third and last part of the volume shows that the debates at the UN 
level have a while to go before they will be able to meaningfully address the 
intersection between AI technology and conflict in cyberspace. There are 
several reasons for that, which are related to the technical/operational and 
strategic/geopolitical perspectives outlined earlier. For one thing, most of 
the richer and top-tier (cyber) military states are often reluctant to forego 
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new military possibilities that may turn out to be game changers.*? Coun- 


tries like the United States, Israel and Russia, which are actively developing 
LAWS are dragging their feet in the GGE negotiations. History does not 
provide much evidence of weapons being banned before they are used. Also, 
politically the level of trust between some of the main negotiating parties 
is at a low point at this moment. The United States are increasingly in an 
adversarial competition with China — which is one of the main contenders 
for the ‘AI crown’ — and since the Russian invasion of Ukraine many states 
are actively trying to sanction and isolate the Russian Federation. These are 
not ideal circumstances to discuss restraint as a governance mechanism when 
it comes to new military and cyber technology. Lastly, there is a mandate 
mismatch between the two UN processes. The UN GGE on LAWS — as the 
name indicates — explicitly focuses on a specific technology (AI) in relation to 
weapons. The UN GGE on cybersecurity focuses on state behaviour as the focal 
point for its recommendations and usually aims to be as technology neutral 
as possible. If a bridge is to be built between these processes it will have to be 
built on sound reasoning on how technology impacts on, or changes, state 
behaviour in cyber conflict. Questions like whether ‘state control’ only exists 
when there is meaningful human control or also exist when in case of ‘system 
control,’ and whether automated and/or autonomous cyber-attacks are or can 
be (in)discriminate*! are likely to be at the heart of that. In other words, only 
by understanding the relationship between AI and conflict in cyberspace as 
a comprehensive phenomenon, and embedded in broader geopolitical con- 
flicts, can the international community truly move forward with meaningful 
regulation. 
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2 The unknowable conflict 


Tracing AI, recognition, and the 


death of the (human) loop 


Andrew C. Dwyer 


Situating unknowability 


From drones, cybersecurity, to robotics and beyond, international conflict 
is intricately intertwined with computation, articulating norms that increas- 
ingly consist of calculation rather than social negotiation.! As computation 
arguably unsettles the dominance of social norms and decision across our so- 
cieties, from social welfare, policing, border points to social media, these so- 
cial negotiations are ever-more intensively (re)asserted as central in attempts 
to limit undesirable” outputs from ‘artificial intelligence’ (AI) systems.” The 
‘human in the loop’ thus steps in to assert authority and control as an exte- 
rior, and rational, residue of human-centric decision that can be drawn upon 
to restrain AI in international conflict and elsewhere.* This chapter argues 
that a dependency by governments, militaries, and corporations on socio- 
normative decision through the human in the loop must be questioned, when 
such socio-normativity is being transformed, modified, and reconstituted by 
the recursive calculability of computation. By this, I mean the ability of com- 
putation to iteratively reassess and identify new, calculative, connections and 
relations. This has implications not only for the outputs of contemporary Al 
systems but also for how international conflict unfolds and its ethico-political 
accounts of practice, control, and authority. _ 

Voracious debate has ensued across academia and policy, from AI Ethics? 
to autonomous weapons systems.° Almost all experiment with the human as 
part of a conditioning loop of control and ethical restraint for AI systems. In 
more nuanced readings, humans are situated at multiple points of decision 
to permit intervention at crucial points in time — as events — where norms, 
responsibility, and ethics can be located and rendered accountable. How- 
ever, as I shall argue, as machine learning algorithms and other forms of 
‘automated’ and ‘autonomous’ technologies become commonplace, there is 
a receding possibility for appropriate translations, or ‘explainability.’’ This is 
because AI systems work on an alternative form of calculative norm-making 
that is not aligned to human commensurability through their architectural 
and logical modes of (re)cognition. My claim is that, due to this change in 
how norms are formed through and with computation, international conflict 
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progressively becomes quintessentially unknowable. It is unknowable where 
conflict takes place, how it is recognised by computation, and who should take 
certain actions. 

In exploring AI systems and unknowability in contemporary, and probable 
future, conflict, I probe “deep! reinforcement learning (RL). Deep RL can be 
summarised, for now, as an iterative system that includes ‘agents’ who ‘learn’ 
from an environment to improve their next action to attain a (pre-deter- 
mined and desired) goal. I do this as a speculative exercise due to its strong 
applicability to adversarial moves by ‘agents’ to examine their potential ap- 
plication in both military wargaming as well as in offensive cyber operations. 
In doing so, I use these cases to raise questions over the neglected role of Al 
on international conflict beyond physical violence and lethality. Their appli- 
cation to conflict, in contrast to robotics for instance, may be understood as 
‘distanced’ from the frontline in the case of military wargaming or unlikely 
to lead to a lethal outcome? in the case of offensive cyber operations. I will 
instead argue that, due to the way that norms are produced through the cog- 
nitive practices of reading, interpretation and action by computation and AI 
systems, imperceptible changes occur not only in how these become prac- 
ticed but also how they are themselves formed and become optimised within 
the contours of international conflict. 

Unknowability is, of course, by no means new; there have always been 
obscurities, where technologies have offered the promise to illuminate the 
spaces of conflict, only this time through big data and algorithmic process- 
ing. Computation, I shall contend however, is not simply another technol- 
ogy,” but actively takes part in forming what the threat, the adversary, and 
indeed what norms come to be established. AI systems are full of doubt, as 
much as Clausewitz noted this with regards to war,” in how they readily pro- 
vide a ‘solution’ that embeds many assumptions and connections unknown 
through both their construction and their ‘learning data.’!! Drawing on 
Michel Foucault’s inversion of Clausewitz “that politics is the continuation of 
war by other means,”!? AI systems offer new ethico-political formations that 
condition discussions on ‘grey zone’ warfare, as well as challenging notions 
of where conflict is conducted and what the international means as it disrupts 
and bends its spatial attributes across our societies. This chapter then outlines 
the ‘unknowable conflict’ that is made present through, and by, AI systems 
through an exploration of deep RL. 

By questioning a reliance of decision — not as a claim that humans have 
no responsibility, but rather shared and complicated by computational (re)cog- 
nition and choice, I thus: (1) outline how technology and conflict has been 
predicated on the notion of the tool which can be controlled and rendered 
knowable; (2) detail how computational capacities for (re)cognition permit 
certain choices that are incommensurable to social negotiation and norms; 
(3) delve into ‘deep’ RL using military wargaming and offensive cyber oper- 
ations as exemplars to consider how they may transform conflict; (4) this then 
informs a discussion on the ethico-political implications of tracing conflict 
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when computational recognition diverges from our knowability, and the 
consequences of conflict amid the loss of the face of the other; before (5) con- 
cluding that the human in the loop is unsuitable for ethico-political accounts 
of AI systems, where computation is transforming norms and practices, and 
in the process decomposing the notions of international conflict itself. 


Technologies and control 


Conflict has always been dependent on various non-digital technologies, 
only more recently have computation and AI systems introduced recursive 
capacities. These challenge conventional notions of an authoritative human 
and a subservient technological tool. This has worked on embedded assump- 
tions of technology as lacking in complexity, as inherently deconstructable, 
and therefore knowable, compared to the sophisticated impenetrability of 
human thought. Yet, since at least the 1950s, an equivalence has been made 
between computation and human cognition and intelligence (such as through 
the neuroscientific inspiration for contemporary machine learning architec- 
tures!%. This has led to an uneasy comparison between our intelligence as 
able to be usurped by ‘machines’ but one that cannot be socially nor politi- 
cally equivalent. This has animated cybernetic debates over automation and 
warfare.’ Yet, AI systems can be considered neither ‘artificial’ nor ‘intelli- 
gent’ but rather something distinct and alternate, as I shall explore in greater 
depth in the next section. AI systems have disturbed dominant perspectives 
between human control, agency, and intent over computation as tool as they 
have grown in recursive capacity.'© AI systems may perform in ‘unexpected’ 
ways, without direct human oversight, or not be easily rendered knowable 
to decision-makers and their norms, but still be based on a wholly logical 
outcome of their prepositions.” In this chapter, I do not dwell on the divide 
between automation and autonomy, S but rather emphasise computational 
forms of recognition and its impact. 

Control, likewise, has been a persistent theme in relation to technology 
and conflict, which has emerged most substantially in discussions on the 
potential ‘autonomy’ of lethal decisions of autonomous weapons systems 
(LAWS).!? Most academic and popular writing argues that lethal capabilities 
should not be performed by computation without explicit human author- 
isation. To address this within militaries, at least within the Anglosphere, 
“human-machine teaming”? and ‘centaurs’?! have been proposed to mix the 
processing capabilities of AI with human oversight, that keep the human in 
or on the loop to varying degrees. Such a position has been criticised over 
the possibility to have ‘meaningful’ control over LAWS, when there may be 
not enough information or time to consider a response.” Such concerns over 
lethal capabilities were most recently brought to attention in a UN report on 
the use of drones in Libya. However, control based on the loop of the human 
as in, on, or even out, of some form of recursive and neat diagram is based 
on a certain understanding to technology and the role of human control in 
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such situations. That is, a human that can rationally execute their decision and 
judgement, either through the code that they write, the operators who control 
the technology, or strategic decision-makers who assess the benefits of certain 
engagements. 

This position assumes a technological tool that is knowable, yet with AI 
systems, the construction, architecture, and learning data bring together new 
formations that challenge such a knowability of the context. Technologi- 
cal assessments of control and authority often present a one way or cyclical 
flow of intent from human to Al system, where there is an “outside to the 
algorithm — an accountable human subject who is the locus of responsibil- 
ity, the source of a code of conduct with which algorithms must comply.”2* 
Such perspectives, as political geographer Louise Amoore argues, stem from a 
knowing, responsible, accountable human. Whereas prior technologies, such 
as non-computational nuclear weapons, may exhibit roughly deterministic 
outcomes, many AI systems are recursive and non-linear, and thus demand 
different attention.” As practitioner Keith Dear says, contemporary com- 
putation has the capacity for a “deeper recursive reasoning that humans are 
capable of, spotting patterns that humans cannot see, and which are beyond 
narrative description, without the encumbrance of human emotional and 
cognitive limitations.”*° Likewise, media theorists have argued for some time 
that there is no direct linear connection between highly recursive systems 
and the ‘performance’ of software.”’ There thus appears to be a growing 
convergence of diverse disciplinary and practitioner perspectives over the dif- 
ficulty of control, and the performative qualities, of computation. 

Although AI does not represent “some kind of epistemological break with 
past modalities ... it reanimates and extends a longstanding intertwining of 
the practices and techniques of security with computational processes,”78 
however, recursion does represent one important change. Technological ra- 
tionalities of conflict based on control, knowability, and authority have been 
gradually eroded by computation, and more recently through the recursive 
cognitive capacities of AI systems. Yet, anthropomorphic narratives of AI 
remain, with words such as “smart, intelligent, sentient, and thinking”? permit- 
ting a particular resonance to human cognitivist comparison. Computation, 
in this reading, operates in a similar way to our forms of intelligence. This 
collapses different forms of recognition between computation and social ne- 
gotiation, so that the former can be understood as simply an automation (or 
even the mind as a computer).°” This leaves AI systems sitting in an awkward 
position between being able, because they are calculative and logical, to be 
controlled, and yet precisely because of their recursion, they are compared to 
the human mind. 


Recognising conflict 


As I have discussed, AI systems have disrupted notions of control, primarily 
through the role of recursivity. Such a recursivity is built on an increasing 
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abstraction from digital binary upon which computation is built.5! As N. 
Katherine Hayles writes in Unthought,>? we are increasingly mediated through 
‘cognitive assemblages’ which imbricate us with computation in ways that 
may not be immediately evident. This is because computation has a capacity 
for choice-making through its cognitive capacities that permits it to read, in- 
terpret, and crucially act, upon signs. This can be seen in the basic ‘reading’ of 
code across different programming languages, interpreting this calculatively 
and through logical axioms, and then taking an action based on this inter- 
pretation. This can appear very limited at the ‘lower’ levels of computation, 
such as at binary logic gates, and in tightly defined programming, yet once 
recursivity, approximation, and probability come into the picture, the choices 
available to computation through its recursive recognition grow. In a similar 
vein, we can understand that this recursivity has permitted a move from de- 
ductive to inductive logic that AI systems work upon, which Luciana Parisi 
summarises as abduction, which is a “speculative dimension of reasoning.”*? 
Thus, this chapter engages in a similar speculative exercise to explore the 
implications of deep RL for international conflict. 

Computational choice-making is thus not made on the same basis as hu- 
mans. It is an alternative form of abstraction through digital binary that is 
reconstituted into higher-level recursive connections and relations. Philoso- 
pher Beatrice Fazi argues that this leads to a quintessential incommensurabil- 
ity between these two forms of recognition.** This is because humans cannot 
adequately recognise and translate the sheer complexity of the high dimen- 
sional relations in deep RL, with thousands, if not more, potential options. 
Computational choices, of course, are not equivalent to the highly reflexive 
decisions that humans make. Yet, deep RL cannot abstract and translate the 
immense possibilities of all the different options, leading to outputs that are 
often singular but arrived at from numerous data points that Amoore articu- 
lates as an algorithmic aperture, which is “an opening that is simultaneously 
a narrowing, a closure.”°? Although computational technologies have been 
integrated into conflict for several decades, ‘automation’ has posed fewer 
concerns as it did not appear to substantively diverge from human intent and 
control. It is only with greater abstraction and recursivity that has brought it 
to greater attention, but choice has always been a condition of computation. 
It is only with AI systems today that this cannot be discarded as ‘error’ or 
‘environmental complexity.’ 

To deal with incommensurability of computational recognition and choice, 
governments, militaries, and corporations have therefore sought to locate the 
human in the loop as a resolution to the problem. In deep RL, one could argue 
that the process of setting ofa goal for an artificial agent means that they adhere 
to ‘social norms.’ However, within deep RL, there is significant interpretation 
by computation to approximate values and optimise towards a goal. Although 
the methods may appear to be zoning in on social norms, computation does 
not recognise that an action may be considered unethical, be good, bad, or any 
other normative standard that may be understood by humans. Computation’s 


24 Andrew C. Dwyer 


choices do not explicitly align with social norms, although they often can, 
producing outputs that are not intuitive to those who are creating strategy 
or launching an offensive cyber operation. Conflict in this sense could then 
become reliant, through computational recognition and choice, on approxi- 
mation, probabilities, and inductive methods of sensing the world. This is not 
to say that the loop could not come in and ask critical questions over such a 
position, but it is about the more insidious and subtle transformations that may 
occur. It could slowly transform what is considered as competent behaviour in 
military organisations, or indeed, as more ‘effective’ and ‘cost-efficient.’ 

Indeed, similar methods are being pursued by militaries — such as through 
agent-based modelling in the UK — to help shape strategic decision-making. 
It does not take a leap of imagination that in the future, if deep RL is substan- 
tively developed, vulnerability scanning, new investments, and new strategy 
and operations will be carried out through computational optimisation. Yet, 
even if a ‘meaningful’ human is present, computation 1s still making choices 
on big data in deep RL through an approximation of rewards in an environ- 
ment. This raises serious questions over the power of such choices and how 
they intersect with decisions made by computation in the future. As Such- 
man, Follis, and Weber argue, this could further contribute to a predictive 
technoscience.°*° This could have damaging effects on potential escalation as 
militaries intersect with multiple other AI systems. Thus, RL may expand 
conflict as it simultaneously retreats to cloud servers away from conventional 
spaces of conflict. As Mackenzie and other social theorists have contended,” 
there is not a simple way with which humans recognise the world, process 
it, and then apply this to computation. Instead, there is a complex mediation 
between multiple humans and computers — cognitive assemblages — which 
mean it is not easy to say how conflict will morph in the future as greater 
capacities for recognition are provided to computation and AI systems. 

The loop (human-machine teaming and centaurs), as I explored previ- 
ously, can be considered as posthuman subjects that are grafted on to a liberal 
notion of humans as atomistic and rational beings.°*® Instead, there must be an 
emphasis on shared notions of action, thinking, and decision that arrive with 
computation and more extensively in architectures of international conflict. 
As Elke Schwarz notes in discussions on the role of LAWS, there must be a 
consideration of 


autonomous weapons not as discrete devices programmed to produce 
a pre-determined outcome, but as cognitive assemblages, a complex of 
sensors, information networks, transmitters, and hardware, which offer 
affordances, together with various human designers, and operators, that 
shape our very ideas of what it means to exert moral agency." 


This suggests that recognising conflict is not simply one that belongs to us 
alone. Rather, conflict is shared and built together in ways that rely on Al 
systems to articulate and render options anew. 
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As Tim Stevens reflects, 


[t]he disconnect between humans and the speed of networked computing 
machines means that the absolute speeds of communication can never 
truly be known to the unaided observer and leads to ever-greater reli- 
ance on computers as the providers of security.* 


This speed afforded by AI is now also complicated by greater recursive capac- 
ities afforded by computational recognition and its choices. This recognition 
and choice-making by computation is however not working in discrete ways, 
as in one algorithm to a human operator. AI systems are interacting with 
one another, creating whole cognitive ecologies that further increase the in- 
commensurability of the future of conflict. This must be processed within 
bureaucracies, organisations, and how they develop AI systems and integrate 
these in military platforms. “! This may permit greater insight, greater forms 
of recognition, but not ones that may be sensible to our knowledge, practices, 
and ways of living. Greater recursive recognition, albeit perhaps unintui- 
tively, is generating a lesser form of awareness; the fog of war grows denser. 


Deep reinforcement learning 


Before considering at greater depth the implications of computational forms 
of recognition upon decisions, norms, and the ethico-political status of con- 
flict, I here develop an empirical appreciation of ‘deep’ RL across military 
wargaming and offensive cyber operations. RL is considered as one branch 
of machine learning in addition to two others — supervised and unsuper- 
vised learning. In supervised learning ‘features are labelled and defined by 
the designer, permitting greater (human and social) categorisation, whereas 
in unsupervised learning, such features are chosen by the algorithm with- 
out explicit human intervention (although with significant influence on the 
weights of the model). In the simplest of terms, if there is a set of photographs 
of cats, supervised learning will be given features with labelled data to make 
connections, whereas in unsupervised learning, the algorithm will create its 
own features to identify cats without labelled data.* These different forms — 
supervised, unsupervised, and RL — have come to be collectively understood 
as machine learning (algorithms). 

Machine learning has become dominant in recent advances in computer 
science due to its significant capacities to recognise the world through re- 
cursive methods as I have substantively developed. As leading computer sci- 
entist Yann Le Cun writes in Quand la machine apprend [When the machine 
learns], the emergence of artificial neural network algorithms, in particular, 
have been key to permitting complex tasks of recognition to be conducted, 
ranging from object recognition, to enabling autonomous vehicles, through 
to natural language processing.” Such methods have permitted a complex 
threading of ‘big’ data that appear to offer the capacity for reasoning and 
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decision-making. As philosopher Yuk Hui details, “[c]ontrary to automation 
considered as a form of repetition, recursion is an automation that is con- 
sidered to be a genesis of the algorithm’s capacity for self-positing and self- 
realization.”*4 Machine learning algorithms then appear to replicate some of 
the useful qualities of humans, recursively analysing data, and in many cases 
do this with a capacity that far exceeds human capabilities. 

RL algorithms meanwhile came to popular attention when Google’s 
DeepMind revealed AlphaGo Zero that “is no longer constrained by the limits 
of human knowledge.” This could beat expert players of the game Go by 
offering a generalised method for beating players of many other games.*® This 
was dependent on a then-new method, ‘Deep Q-Networks’ (DQNs), that 
combine RL with unsupervised machine learning — in this case convolutional 
neural networks.” RL can be most simply expressed as composed of an agent 
that optimises its actions to gain a reward within an environment through trial 
and error. Crucial is the generation of an appropriate value for the reward to 
permit ‘new’ experimental actions, as well as maintaining a record and reward 
signal of what has worked in the past. The aim of RL is to improve a policy so 
that it can maximise the reward for the agent, and iteratively and recursively 
improve its strategy. As one computer science text notes, “an agent must be 
able to learn from its own experience.”** That means that an agent can re- 
cursively iterate what may be the optimal strategy to achieve a defined goal. 

However, in RL, there are limits to the number of different potential 
environmental ‘states’ that can be processed, which leads to the ‘curse of di- 
mensionality.* This is where there are too many states to be processed, with 
higher dimensionalities (or simply connections) between data, which make 
it computationally infeasible in both processing power and time. Another 
difficulty is providing a reward that both appreciates short-term gain along- 
side long-term exploration of different potential actions. When “large” prob- 
lems are addressed (as in the case of conflict, with its many environments 
and adversary actions), it becomes infeasible to rely on hard-coded reward 
scenarios for environments that provide a very high number of potential 
options. Thus, DeepMind’s DQNs offer a capacity to approximate a reward 
using “experience replay’ to allow an agent to experiment and optimise their 
actions. Deep RL “uses neural networks to represent the policy and/or the 
value function, which can then approximate arbitrary functions and process 
complex inputs.”°” For conflict scenarios, the use of deep RL offers the 
promise to bridge “the divide between high-dimensional sensory inputs and 
actions.”°! Yet, as deep RL grows in its recursive and abstractive capacities 
through calculative cognition, the more it must rely on approximation and 
probability. 

There are many different flavours of deep RL. This can include those 
using a predictive model and those that are model-free (where the learning 
algorithm develops a control policy directly) or have different methods to 
develop ‘self-play’ to train their agents and policies. Indeed, supervised and 
unsupervised learning “do not provide a principle for action selection and 
therefore cannot in isolation be enough for goal-orientated intelligence”5? 
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unlike RL. This emphasis on goal orientation is purported by those who 
work in the area as key to more ‘general’ AI and the benefits for goal-oriented 
machine learning are clearly of interest to militaries. The capacity for action 
selection, along with possibilities for multi-agent RL, has led to its applica- 
tion in autonomous air traffic control? and predator avoidance in robotic 
systems.5* There are also connections between game theory and deep RL? 
that make it applicable to a wide range of applications, including military 
wargaming and offensive cyber operations. 


Wargaming for conflict 


Wargaming has a long and complex history,°° has been widely applied to 
cyber conflict,S7 as well as in assessments of how AI systems, including deep 
RL, can inform such practices. °° Due to the capacity of deep RL to of- 
fer many environmental states and goal-driven agents, they appear to be 
particularly suitable to wargaming. Unlike a focus on ‘human-machine 
teaming? and other hybrid forms of human decision making in conflict in 
the UK and USA, China anticipates that “AI will accelerate the process of 
military transformation, causing fundamental changes to military units’ 
programming, operational styles, equipment systems and models of combat 
power generation.” ®? The drive to work with AI systems in military settings, 
such as thinking around the role of ‘improving’ planning and strategy, could 
use deep RL to test different scenarios and adversary actions. 

Deep RL could deeply inform how conflict is simulated and the forms of 
normative, and calculative, knowledge that are produced. Even today, ‘agent- 
based’ modelling“ is central to UK Strategic Command’s Digital Strategy for 
Defence.€! This includes continued funding for a Single Synthetic Environ- 
ment procured from the UK business, Improbable,°” with a claimed ability 
for the UK to model responses to potential conflict scenarios. Although it is 
unclear from public statements whether (deep) RL plays a part, it is likely to 
incorporate at least some parts of an AI system. Likewise, researchers from 
Tianjin University in China have been experimenting with deep RL to aid in 
wargaming,°* demonstrating that these forms of AI systems are already being 
tested for their application to such activities. 

In a report commissioned for the UK Defence Science and Technology 
Laboratory (Dtsl), there is an extensive reflection on the capacity for the in- 
tegration of deep RL into wargaming.” The report claims that “there is little 
work showing good performance with Deep RL on unseen environments ... 
[t]his does not mean Deep RL is not useful in the context of wargames, but is 
not suitable for completely unseen scenarios.”®° This is affirmed by computer 
scientist Le Cun, who notes that deep RL 


simulators have to be powerful and precise enough, that is to say, they 
must reflect what is happening in reality enough so that, once the system 
is simulated, its capability can be transferred to the real world. This is not 
always possible.°° 


28 Andrew C. Dwyer 


Although today deep RL may not be applied well in ‘unseen environments,’ 
this may be becoming less of a limiting factor. In 2019, DeepMind developed 
a multi-agent deep RL, AlphaStar, to play at human ‘grandmaster’ levels in the 
complex strategy game StarCraft 11.8’ Due to the limited observations from 
the viewpoint of the agent in StarCraft II, as well as unseen environments 
and adversary movements, this is distinct to the bounded state solutions of 
AlphaGo.55 The authors argue that “many applications have complex strategy 
spaces that contain cycles or hard exploration landscapes, and agents may 
encounter unexpected strategies or complex edge cases when deployed in the 
real world.” For wargaming, this ability to work with a partiality of infor- 
mation and obscured environments is important. Yet, as Goodman, Risi, and 
Lucas note, various deep RL are unlikely to be used in planning for opera- 
tions but they could be useful for concept and force developments due to their 
“explicitly exploratory goals.””° 

Although it is unlikely that deep RL will be used for conflict operations 
at least in the near future, its exploratory capacity could be adopted for strat- 
egy in ways that are more advanced, yet recursive, than agent-based model- 
ling. The capacity to model adversary behaviour could shift how militaries 
understand investment decisions, strategic planning, as well as the potential 
future possibilities of adversary behaviour. Although individuals may dispute 
certain outcomes of deep RL, military bureaucracies and other agencies may 
be tempted to subtly rely on such outputs. It is well documented that mil- 
itary personnel typically trust the outputs of computation, at the very least 
in time critical situations. However, it is my argument that such recursive 
techniques of deep RL could transform what is considered normal in con- 
flict, and actively shape how environments and procurements are made. It 
is not about individual, rational decision makers, but how such practices of 
deep RL, may shape organisational cultures, slowly revolutionising the prac- 
tice of strategy. Changes may be imperceptible on a case-by-case basis but 
relying on deep RL's approximation of behaviour leads to an optimisation of 
conflict itself, with ethico-political consequences that could embolden stra- 
tegic actions or prioritise subtle manoeuvres, each with potentially worrying 
complications. 


Offensive cyber operations 


Unlike in wargaming, the application of deep RL may be more difficult 
in operational ‘kinetic’ settings — such as with drones and robotics — due to 
the large range of unseen and data-poor environments.’” Offensive cyber 
operations, by contrast, may be more readily adopted due to their lack of 
connection with lethality and therefore making it a space of ‘innovation.’ In 
Automating Cyber Operations, Buchanan et al. note how RL “could someday 
be a game-changer for offensive cyber operations,” ° despite much contem- 
porary work focusing on its application to defensive methods, such as pro- 
tecting the ‘Internet of Things’ and improving intrusion detection systems.” 
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One development however includes Baillie et al.’s prototype autonomous 
cyber operations ‘gym’ that tests operations across ‘blue’ and ‘red’ adversarial 
teams.” They claim that it “aims to provide a scalable, efficient and flexible 
training environment that uses a high fidelity emulation and lower fidelity 
simulation to facilitate RL for the coevolution of red and blue agents capable 
of executing cyber operations.”’° By combining simulations (to train agents) 
and emulations (to adapt these to ‘real life’ scenarios), these are important 
developments that may offer a foundation to recursively enable AI systems to 
mount cyber-attacks in the future. This could be done by exploiting certain 
vulnerabilities, and other potential attack vectors to enable greater offensive 
cyber operations that permit operational engagement either directly, or by 
informing new methods that may be used. 

Some further applications could include the adaption of deep RL to con- 
ventional ‘penetration’ testing!” to identify vulnerabilities in computational 
systems. Although this could benefit defence, it could permit offensive pivots 
on networks, such as through the tool DeepExploit,’* or to even exploit dif- 
ferent elements of the cyber ‘kill chain.’ This could have significant effects 
on cyber operations, which could allow bespoke, highly obfuscated attacks 
from unusual vectors, or to generate new perspectives on vulnerabilities. Yet, 
as is common with other applications of deep RL, there are significant issues 
on the complexity of computational systems and their ecologies. Deep RL ar- 
guably optimises better in ‘data rich’ environments, so as states such as the US 
move towards persistent engagement and aim to employ stealthy measures to 
gain strategic advantage,®” the risk of using malware with deep RL may be 
too risky, for example. Even slight ecological changes for a deep RL agent 
could be troublesome, which may lead to its application in more, perhaps 
‘indiscriminate,’ widespread attacks which are not dependent on stealth — 
meaning its impact on state offensive cyber operations will remain incon- 
clusive. This is particularly so as the ethics of offensive cyber increasingly 
considers the impact of AI on operations.5! Questions will also be raised 
on how this could fit into broader ethical, legal and operational aspects of 
emerging state organisations for offensive cyber operations, such as the UK’s 
National Cyber Force.*? 


Environments 


Deep RL has the capacity to be applied to a range of conflict scenarios, 
such as those highlighted above. Although advances have been made since 
AlphaGo Zero in 2017,85 such as with AlphaStar discussed above, there are 
persistent issues of data availability when in conflict situations and complex 
environmental and ecological dynamics. Yet as Lee et al. note, even in cases 
where there is a partiality of data and knowledge of the environment, agents 
performed better than humans in StarCraft ii Regardless, deep RL is still 
primarily suited to ‘structured problems’ which allow for an optimal strategy 
to be converged upon for full game simulations. That is, adaptions to games 
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will not easily lend themselves to real-world adversarial engagements.55 As is 
reflected on reports of the promise of AI to cyber conflict, deep) RL is not 
likely to be useful in the immediate term.80 So, the rest of this chapter reflects 
on how deep RL could contribute to a changing dynamic of conflict, where 
decisions may still appear to be made by humans but which are actually being 
built in co-constitutive ways. 

Although the work on StarCraft II by DeepMind is a “well recognized new 
milestone for AI research, [StarCraft II] continues to present a grand challenge 
to deep RL due to its complex visual input, large action space, imperfect in- 
formation, and long horizon.”55 However, even more recent developments by 
DeepMind on MuZero combine different algorithms into a broader AI system 
that “does not require any knowledge of the environment dynamics, poten- 
tially paving the way towards the application of powerful learning and planning 
methods to a host of real-world domains from which there exists no perfect 
simulator.”*? This demonstrates the fast pace of change, and for how militaries 
and others may quickly pivot and adapt to the potential of these technologies. 

Environments will remain an essential and limiting factor to their deploy- 
ment, in understanding their dynamics, and how ‘agents’ will work within 
them. Any computational representation of an environment is an abstraction, 
and thus there is always within it the potential of surprise and difference. 
Within conflict, environments are crucial, so that we could say that deep 
RL is not just about agents and adversarial behaviours but also a new form 
of battlespace, a space that retreats, and reconstitutes conflict itself. In so 
doing, the security of such environments is paramount, as they are likely to 
be targets if they are used for strategy or even offensive cyber operations.” 
Such environments could be subtly manipulated by adversarial inputs that are 
imperceptible to humans, where computational recognition is changed just 
enough to transform the outcome of a strategy or operation beyond a hu- 
man capacity to intervene. This can include adversarial attempts against deep 
RL with limited knowledge (black box) to those with complete knowledge 
(white box) with a variety of different methods, such as data poisoning.”! 
Thus, here we can see that deep RL will not ‘solve’ conflict but may appear 
to offer new perspectives where agents act through approximation and opti- 
misation, introducing greater risks to the ‘control’ of conflict. Such attacks, 
where incommensurability is a condition of its performance, then add yet 
another imperceptible trace of unknowability in their ethico-political standing. 


Tracing conflict 


The emergence of technological autonomy linked to advances in AI 
and represented by the growing research interest in algorithms intro- 
duces a novel, potentially game-changing element for the mature models 
of human agency and material/nonmaterial structure dominant in IR 
theory.” 
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As Hendrik Huelss reflects on norms and algorithms in international rela- 
tions, AI systems have the capacity to radically challenge how human agency 
and computation come together. As computation has a capacity for choice- 
making, this challenges conventional technological thinking as it actively 
constructs what is optimal and how conflict may be practiced, albeit inter- 
sected with human decision-making, as a specific form of reflexive choice. 
Yet, as I have hinted towards, there are imperceptible connections, or traces, 
that lead to new formulations when conflict is unknowable in relation to 
deep RL. In this section, I shall outline how traces could help understand 
the complex interplay across incommensurability for both offensive cyber 
operations and strategic wargaming. 

AL systems are reliant on big data to generate their own ‘ground truths,’ 
but this is often hosted through cloud computing servers, distributed and 
distanced from the ‘face’ of conflict. Conflict has conventionally occurred 
over there, but as with other forms of security these have been brought into 
the everyday, where processing of data and actions are made at great distances 
from where they may be performed, such as in drone warfare.” With deep 
RL, it is not only about where cloud computing and algorithmic processing are 
occurring but also that new spaces and notions of conflict are being created 
by and within deep RL. By this, I mean the data and assumptions for environ- 
ments, the possible actions of an agent, and their optimised and approximated 
rewards, are articulating new worlds and possibilities. International conflict, 
as such, could be actively performed in the spaces of the adversarial relation- 
ships of deep RL, where computational choices, human decisions, designers, 
environments, big data and more come together to optimise the future. Some 
of these futures may become understood as more probable, more likely, but 
only in conversation with these others. These transformations could lead to 
more subtle changes in the norms of international conflict, and its traces far 
more complex than could be immediately imagined. 

One method to exemplify tracing conflict is turning attention to the neu- 
rons that form the ‘deep’ part of deep RL, the artificial neural networks. 
Some methods are seeking to expose how to ‘reveal’ how a neural network 
may have ‘seen’ features, especially in image recognition.” However, I argue, 
along with Fazi,” that there is an incommensurability here, so that even if 
values partially represent choices made by computation, there will always be 
an unknowability beyond representational critique, a trace of another recog- 
nition. In simpler terms, there is no adequate translation available from deep 
RL to human social negotiation or norms. In Figure 2.1, I present a very ab- 
stracted (cybernetic and neuroscience inspired) representation of a neuron as 
is common in computer science literature. On the left, arrows point towards 
the circle. The arrows each have a weight w, that are calculated with a ‘learnt’ 
bias > that then produces an output y4. Input weights may emerge from pre- 
vious neurons in neural layers, that each output an ‘activation’ with its own 
weight to pass on to the next layer. There are many different variations of this 
process, which ‘layer’ it is in, and whether certain neurons are activated at all. 
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Figure 2.1 A simplified diagrammatic of an artificial neural network algorithm neu- 
ron, as popularised through cybernetic comparison. 


Source: Author’s creation. 


Here, we can sense the traces of such a process, but never intuitively feel or 
comprehend wholly what these may be. 

Even if multiple options are presented in the output of the neural network 
algorithm for the reward value, one cannot wholly excavate why such an 
outcome of the loss function was made. It may be possible to look at the ac- 
tivation points of a neuron, but it would be difficult to assess how this then 
comes to interact across the environment, the agent, and the policy — where 
computational choices intersect with one another so that it may be become 
unclear what is exactly leading to a strategic offer or the best method to target 
an adversary. This is only magnified when there are multiple systems. Cru- 
cially, neurons operate to condense possibility and must have bias (intended 
or unintended) to function.” Thus, they must condense all the possibility, 
and in the case of deep RL, approximate a reward. To trace conflict is not to 
identify points of activation but to incorporate contexts from big data, their 
biases, and how these may be used in determining an environment for the 
agent. If data is used from datasets outside previous spaces of conflict, then 
conflict becomes intertwined with these, where they become unknowingly 
implicated into the practice of conflict itself. 

Tracing conflict is not just about the isolation of neurons within deep 
RL — or other AI systems — away from the big data, environments, and ecol- 
ogies we live in, but the way AI systems wrap up contexts and previous 
weights of human decisions and computational choices in complex hybrids. 
This argument can work alongside the good work that is ongoing to identify 
biases within learning data or to show potential points. However, this will 
always be insufficient to address the complexity of the traces that emerge in 
assessing conflict. This is one of the trickiest of puzzles. It is not only that 
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there is not enough data, particularly in conflict situations, for there to be ro- 
bust assessments of how AI will react?” but also that the greater the volume 
of data, the more unknown conflict becomes. Thus, both too much and too 
little data, amid the impossibility of ever truly ‘simulating’ an environment, 
suggests a persistent unknowability that emerges with AI systems. 

Therefore, to even consider deploying highly recursive systems is not about 
the ‘events’ or points of decision of the human in the loop, but rather about 
how to incorporate the doubt of this computational unknowability, and to 
assess if it is sensible to involve it in conflict at all. A neural network must 
condense contexts of the past to produce a future output. AI systems col- 
lapse different places, contexts, and knowledge into a particular moment 
where weights are produced to suggest the significance of this or that action. 
Future conflicts with deep RL will be both hyper-local to the learning al- 
gorithms as much as they are yet distributed simultaneously across learning 
data and simulated environments, bending our notions of scale and even 
the international through their traces. When conflict is mediated through 
computational forms of recognition to identify an optimum strategic output, 
the political space for failure, chances for social reflection recede. Where is 
doubt? If actions by deep RL suggest a future action that cannot be under- 
stood now, should that ever be taken? These are crucial ethical and political 
questions that cannot be solved through calculative forms of recognition. As 
Jacques Derrida contends, decisions are not of the order of the calculable.”® 
But decision cannot be ‘inserted’ into the machine learning algorithm as 
ethics precedes a codification. Nor can its outputs be recaptured by the loop 
of the human to somehow reassert authority and control, as computational 
recognition disturbs the knowability and commensurability of the outputs 
themselves. 


Trace as an ethics beyond ‘the loop’ 


We might understand that deep RL performs in multiple spaces and across con- 
texts, even if it is located on a particularly ‘secure’ server within the UK or US 
for instance. Therefore, if multiple militaries adopt deep RL, it is possible to 
understand this as conflict retreating to forms of computation cognition and 
choice. This may appear to be a radical claim, but Chinese strategic thinking 
of an AI singularity,”” as well as a common ‘winner-takes-all’ approach that 
sustains military thinking, point towards this as one potential trajectory. Yet, 
in such practices, conflict conducted by computation should not be considered 
as conflict, when conflict is informed by a humanly impulse, emotion, and 
social sensibility. I take such a position from thinking with Emmanuel Lev- 
inas, and his work on the notion of the face.1% That is, our intrinsic human 
condition is to be in relation to the other before we can constitute ourselves, 
that has also been used by Judith Butler in feminist thinking on violence.'”! 
Although there is not space here to go into the details behind such thinking, 
the importance is that, for Levinas, ethics arrives before its articulation, in 
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relation with the unknown other.'°? In simpler terms, ethics is not something 
that comes after ourselves, but is inherent to us. Ethics cannot thus be cod- 
ified and inserted into deep RL. In offensive cyber operations, if the deep 
RL simply optimises the output with steps that may be unknowable, it is not 
conducting this through a social understanding of norms or ethics. The face 
is not a physical manifestation but affords a capacity for an ethics before it has 
been witnessed the other. When understanding computation, it does not have 
a social recognition of the face, and thus it is indifferent and incapable of ever 
attaining a social ethics of the other. 

A trace according to Jacques Derrida is “impossible-unthinkable-unsayable 
in that it connects us in ways that are not directly traceable. I use this in my read- 
ing of the potential for AI systems to conduct conflict to claim that there 
will never be a wholly ‘explainable AI’ of its multiple traces as much as it 
cannot fully bear the weight of contexts past. In this chapter, then, I do not 
seek to see the trace as something which can be explicitly drawn. This is a 
trace that is unable to be wholly articulated, it is incommensurable. However, 
traces hold us accountable to other, to the face, in our social negotiation of 
norms and ethics. The reason to raise such a point 1s how the face of conflict 
is incommensurable to computation, even as many are frantically trying to 
cement an authoritative human in the loop to stamp an anthropomorphic 
face on deep RL and other AI systems. In wargaming, the trace of the human 
other through our social negotiation is always there, but it is incommensu- 
rable to deep RL. Although, undoubtedly, there are traces of our ethics and 
politics, they get twisted and contorted to the point that our own social ne- 
gotiation may become incommensurable to itself: that is, our ethics morphs 
towards optimisation and means that conflict itself becomes a zero-sum goal. 

As I have said, AI systems do not abstract in a human sensibility of our 
social connections, meaning that my claim is that it is not sensible to consider 
that computation generates a similar ethico-political construct of conflict, as 
much as it abstracts away from, and sterilises the possibility of such conflict. 
As Bergen and Verbeek state, Levinas did not explicitly address technology, 
but they suggest that technology with a ‘face’ has emerged as they have be- 
come ‘humanized.’!°* With AI systems, this is most clear in the cybernetic 
comparison to human intelligence, as if it is on an equivalent plane to com- 
putational recognition and choice. This is not the case, as computation is not 
necessarily on a hierarchy above or beneath us but offers a different way of 
being in the world. The danger is that deep RL is understood to have a face 
through its interfaces and outputs, anthropomorphising computation with a 
trust roughly equivalent to a human ethico-political sensibility. 

So, why focus on such a point of the face for the ethico-political future of 
conflict? As recognition is increasingly performed by computation, its calcu- 
lative logics and choices will become key to articulate the norms and practice 
of conflict. Thus, in a deep RL, the ‘agent’ does not embody the face of 
the other in its recognition. This is because it does not recognise or relate 
on an equivalent basis to us. It is incommensurable to our ethico-political 
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frameworks. It is not only in LAWS that we must focus our attention but also 
on offensive cyber operations, wargaming, and elsewhere that are not immedi- 
ately lethal and on the ‘front’ line. AI systems will condition organisations and 
bureaucracies to plan and operationalise future conflict. It may be that there 
is healthy scepticism of such outcomes of deep RL in military planning, but 
the solution is not to have the loop. As Schwarz details, “the role of the human 
as a moral decision-maker in the technological loop is considerably more 
complex than the picture of the functional and rational agent ‘in’ or ‘on’ the 
loop might suggest.”!°° Deep RL emerges as a product of multiple authors — 
software engineers, learning data, assumptions, decision-makers priorities, 
computational recognition and choice, and more — thus rather than concep- 
tualising the loop, humans and others are already intrinsically, unknowably, 
enmeshed within it through traces. This changes the ethico-political rela- 
tion, when the adversary is no longer simply face to face but rather becomes 
assumed, constructed, and abstracted and partially recognised through traces 
by computation. 

Traces expose the falsity of the loop. So, when it is said ‘is the human there, 
can that human have a ‘meaningful’ say, is to assume that there is a possibil- 
ity for us to translate the face through computation. As conflict is mediated 
and interpreted by AI systems, conflict ceases to be conflict in a meaningful 
sense. Instead, rather than understanding AI systems and conflict as some- 
thing that can regarded as ‘evolutionary’ or ‘revolutionary’ in debates on 
technological change in international relations, °° conflict is beyond, inter 
alia, human knowability, when AI systems grow in application. If it down 
to approximation, calculability, and unknowability, then conflict is trans- 
formed, even if there is an ability to say ‘no’ to this or that particular action — 
such as that cyber-attack or this plan emerging from a wargame. 


Conclusions: unknowability and persistence 


Warfare and conflict are not only unknowable due to a lack of data or the ‘fog 
of war’ but also due to the face that is incommensurable to AI systems as well 
as inadequate abstractions and translations of outputs made though computa- 
tional recognition and choice for humans. In this chapter, I have offered both 
a conceptual and practical reflection on deep RL — through wargaming and 
offensive cyber operations — to detail both a general transformation of conflict 
as well as the particular of the difficulties of operationalising such technolo- 
gies. There is no possibility of an escape from traces in conflict, but compu- 
tation has seriously complicated how these traces lead to an ethico-political 
sensibility. Bellanova et al. argue that there is a “force of computation”!”” that 
perpetuates certain forms of violence and harm, that I think can be applied 
to the incommensurability of a face to AI systems. Across ‘cognitive assem- 
blages, computation’s calculative recursivity and recognition will transform 
how militaries, governments, and corporations understand conflict through 
AI systems. As there is no possibility to codify computation with human 
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sensibilities, this is particularly dangerous, not only for lethal autonomous 
weapons but also for the broader optimisation of conflict in the future. 

I have sought to trace how deep RL transforms how international con- 
flict may be understood. First, I have considered where conflict may be con- 
ducted and have argued that, through deep RL, it will occur through new 
battlespaces that occur both through and within AI systems. This transforms 
what is considered as the spaces of international conflict as multiple deep RL 
agents simulate the adversary, conflict is no longer just on the ‘frontline,’ but 
is withdrawn and retracted, disrupting what the ‘international’ of conflict 
could be. This is amplified by traces that becomes threaded across AI systems 
and humans, which expand the sources of what informs the practice of con- 
flict. Second, I have assessed how conflict is transforming through computa- 
tional cognition and choice. This has demonstrated how recursivity has been 
central to transforming dynamics of control, away from a deterministic tool, 
to one that actively participates in the performance of conflict. In offensive 
cyber operations, this could be ‘agents’ deployed on an adversary network, 
to suggested strategic moves in wargaming that inform how a military or 
government may respond. And third, I have questioned who is performing 
conflict. As computational recognition and choice challenge the commen- 
surability between human and social norms on the one hand and calculative 
norms on the other, conflict becomes distorted. As I have explored through 
the face with Levinas, the norms of what is considered ethical or appropri- 
ate, will be expressed with and through AI systems. This means that the 
social norms of international conflict could be slowly transformed. Thus, AI 
systems introduce an unknowability to international conflict and its perfor- 
mance through cyberspace. That is, it is unknowable exactly (from) where 
conflict is taking place, how conflict is being performed and constructed, and 
who (and what) is doing such work. 

Ultimately, there is no “outside” for a rational human in, on, or out of the 
loop, as we are always imbricated within it. It is not that there is a computa- 
tional choice and human decision maker that exist in a binary divide. Instead, 
they are co-constitutive, part of ecologies of interaction of organisational 
cultures, computational choices, decisions, data, and so on. It is recursivity 
that has become so important — even as computation has been key to the 
enactment of international conflict for many decades. This is what makes 
conflict and AI systems profoundly different, there is no outside with which 
to retrieve an ethics as much as there is no ability to wholly steer it from 
within; AI systems have no one person who has the capacity to know all the 
traces and can exert their explicit intent. Yet more insidiously than the con- 
trol promised by the loop, computation transforms norms through its outputs. 
Therefore, in embracing this unknowability, I claim that there is a danger 
that in the hope of deploying operations with speed and new insights offered 
by computational recognition, there is a loss of the ethical and political face 
of conflict, where optimisation holds sway. So, in advocating for the death of 
the loop, I hope to prevent the slow death of our ethico-political relationship 
to one another. 
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A double-edged sword 
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Generating and detecting misinformation 


There are two complementary approaches in statistical classification for 
generating content using artificial intelligence (AI): the Generative Adversar- 
ial Network (GAN) approach and the Discriminative approach. The goal of 
the generative models is to produce synthetic data points that are randomly 
drawn from a wider distribution like the one responsible for generating real 
data points. For example, a generative model could generate fake photos of 
people that do not exist. Such photos, when analysed by a human or another 
machine learning model, would likely be classified as genuine human faces. 
On the other hand, discriminative models aim to distinguish between dif- 
ferent kinds of data presentations. A discriminative model can be trained, for 
example, to distinguish a photo of a dog from a photo of a cat. 

Generative models are more challenging to build than corresponding dis- 
criminative models. A generative model for images should be able to learn 
correlations like “boats are almost always surrounded by a large body of 
water” and “road bikes are rarely observed near a large pile of snow”. Dis- 
tinguishing cats from dogs seems to be a much easier task to do. Indeed, this 
intuitive discrepancy in difficulty may be explained. Discriminative models 
are supervised learning problems, in which the model is provided with la- 
belled data to learn from, such as photos of cats and dogs divided into two 
corresponding piles. The challenge of building a generative model is that the 
model is expected to produce images that are normally considered as an input 
to traditional supervised learning algorithms, but it is not clear how such a 
process should be guided. A natural idea is to pass the output of a generational 
model to a discriminative model for evaluation. Introduced in 2014, this is 
what the GANs do. Two neural networks are trained at the same time, play- 
ing a zero-sum game in which one algorithm’s gain is the other one’s loss. 
The generative network generates candidates such as cats and dogs, and the 
discriminative network evaluates them. In other words, the generator tries 
to fool the discriminator, which at the same time learns to do its part — that 
is to recognize genuine images from the artificially constructed ones. An 
important property of GANSs is that the networks can learn from each other 
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in an unsupervised way, that is, without providing any new data points such 
as labelled pictures of cats and dogs. As they progress through this iterative 
process, we can observe them improving their ability at a task. 

GANs became very good in the computer vision domain but Natural Lan- 
guage Processing (NLP) problems are much more challenging, including 
generating and recognizing the textual dis- and misinformation. The main 
problem is that while text is a discrete object, as there is only a finite number 
of letters in the alphabet, the set of possible words is enormous. Specifically, 
it is difficult to back propagate the loss of signal from the discriminator back 
to the generator, that is, the discriminator cannot easily communicate with 
the generator to provide feedback on how well it did its job. The reason is 
that the argmax operation used by the generator to produce discrete symbols 
is non-differentiable. There are three ways to solve this issue, use reinforce- 
ment learning strategies, operate on continuous representations, or use the 
Gumbel-softmax operation. 

Some older and less effective approaches used previously, include Naive 
Bayes, Hidden Markov Models (HMMs), and plain Recurrent Neural Net- 
works (RNNs). The current approach to generate synthetic, human-like 
text, however, is to use Large Language Models (LLMs) that are carefully 
pre-trained on enormous datasets. Generative Pre-trained Transformer 3 
(GPT-3) created by OpenAI is a deep learning network with a capacity of 175 
billion parameters, ten times more capacity than the second largest system, 
Microsofts Turing NLG.” The main power of the OpenAI model comes 
from careful pre-training. To compare the scale of this process, the entire 
Wikipedia is part of the data used to train this model but amounts to just 3% 
of the hundreds of billions of words used for training. 

GPT-3 is able to learn rapidly from a small set of training examples due to 
extensive pre-training and parameter size. An example of this would be to 
train GPT-3 to translate documents written in English to French by providing 
a few examples of English to French translations to the model and leaving the 
last example untranslated for GPT-3 to produce the translation. This task does 
not require tuning or weight updates and would not be possible without an 
extensive pre-training process. Its generic architecture can be applied to most 
tasks. It has been demonstrated that GPT-3 is able to learn 3-digit arithmetic 
and is also capable of coding in Python and other programming languages. 
Finally, GPT-3 excels at its primary task, generating text. GPT-3-generated 
texts are nearly indistinguishable from human-written texts. In one experi- 
ment, 80 native speakers were asked to identify which 200-word documents 
are generated by GPT-3 and which ones by humans. The subjects correctly 
distinguished GPT-3 generated texts approximately 52% of the time, which 
is not significantly higher than a random chance. 

Humans interact with computers, cell phones, and other electronic de- 
vices in numerous ways. To attract customers and make their experience 
both natural and pleasant, developers of cell phone applications or computer 
software try to improve human-computer interaction (HCI) by including 
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natural interactions such as voice user interface (VUI) and enabling dialogs 
resembling conversation between people. Models such as GANs and LLMs 
are able to reduce the cognitive barriers between machines and people. 

An early example of reducing cognitive barriers was designing interfaces 
for querying large databases using natural language. In such solutions, the 
user formulates a query for the type of products they are looking for in an 
online store, using common speech. The speech is interpreted into text and 
translated into formal query language. After the query is executed, a reverse 
process is performed. The result is first translated to a natural language sen- 
tence, which, in turn, is emitted as a voice response. A recent and advanced 
development of this kind is a product developed by Microsoft called Power 
Apps. It provides a low-code application development process in which you 
describe, in plain language, what you want to do and an application is quickly 
created.* 

Following our customer of the online shop example, GANs can be used 
to dynamically create images of the offered products in the environment 
where they are naturally used. For example, instead of showing a tradi- 
tionally stand-alone image of some piece of fashion, it can be adjusted to 
customer's needs and presented in various contexts of a person wearing it, 
including matches for gender, height, weight, and skin colour. It is impor- 
tant to highlight that although similar solutions are available without GANs 
they are limited because they are always custom-made and require significant 
involvement of human designers. As a result, they are labour intensive and 
expensive to create. 

Another use of GANs and LLMs focuses on their core capability, the abil- 
ity to generate synthetic objects that may be considered a sample from a 
distribution generating genuine instances. Such an application is described 
where GANs are used to generate synthetic orders that resemble genuine 
ones for e-commerce platforms such as Amazon, Alibaba, or Flipkart. Such a 
tool allows retailers to explore the space of plausible orders, and can provide 
insights into future customer orders, preferences, or price sensitivity.” 

Since LLMs are trained on enormous and unlabelled datasets, they produce 
models that have bias against gender and marginalized groups that are present 
in the digested text. This raises valid criticism that should not be unexpected. 
This was noted in the original paper introducing GPT-3, which includes a 
detailed analysis of the biases observed including gender, race, and religion.® 
It is important to remember that the models are designed and expected to 
reflect the real world rather than how we would like the world to be. It is 
possible to alter the model to reduce bias, potentially making them less reflec- 
tive of the real world and there are approaches to accomplish this.’ GPT-3, 
and its earlier version GPT-2, exhibit bias against Muslims, marginalized 
groups, and other minorities. This is true for other LLMs and pre-trained 
embeddings.’ Occupational-based bias is also known to be present, associat- 
ing women with occupations such as babysitting or nursing, and men with 
occupations such as construction workers and truck drivers. 
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The ingested text will likely include inflammatory or hateful content. As 
a result, LLMs might generate not only biased text but also hateful and ex- 
tremist content]! Unfortunately, it means that GPT-3 might be weaponized 
to create conspiracy, extremist, misleading or hateful content with little in- 
tervention by humans.” It has been suggested by several researchers that the 
datasets used to train these models should be carefully curated to reduce this 
possibility.’ 

Since LLMs try to “memorize” the training datasets, it might be possible 
to extract sensitive information from the model. For instance, with queries 
carefully designed for this purpose, one study extracted private information 
including phone numbers, full names, addresses, social media accounts, and 
more.!* It may also be possible to generate sensitive national security infor- 
mation through a process recognized by intelligence agencies as classification 
by compilation. 

Finally, let us mention that pre-training of LLMs often takes weeks of 
continuous computation done by hundreds of CPUs.!5 As a result, the cost 
to train the network can be estimated to be hundreds of thousands of dollars 
implying a sizable financial cost as well. This concern has been raised and 
some approaches for the optimization of the pre-training process have been 
proposed.!° 

There are several approaches to detect computer-generated text, most of 
them trying to detect style differences in which humans or generative mod- 
els produce text. Such analyses work well on human-written text and can 
be used to detect the author of a specific document. The easiest approach is 
the bag-of-words classifier that computes how often given words are used in 
the document. Slightly more sophisticated approach is the skip-gram model, 
which tries to detect the context in which a given word was used. These 
techniques are only a start and more approaches are needed. As LLMs get 
better in mimicking patterns of humans and ensuring that no suspicious word 
sequence is present this might be used as evidence that the text was not writ- 
ten by a human. 

Algorithms detecting generated text have quickly become ineffective, 
but research in this domain continues.” The state-of-the-art detectors 
GROVER! and RoBERTa” are approximately the same size as GPT-2, 
making it impossible for average researchers to use. It appears that to detect 
computer-generated text, access to computing power comparable to mali- 
cious actors generating the text is necessary. Moreover, it has been demon- 
strated that — at present — a simple statement inversion and other semantically 
neutral modifications can confuse the detector.” The problem of detecting 
generated text is extremely challenging but it may not be the issue that should 
be most concerning. A closely related question is to detect misinformation 
that aims to create a societal harm. This problem may be easier and there may 
be solutions. 

The most natural approach to identifying misinformation is using Knowl- 
edge Graphs (KGs) to build a content-based detector. Unfortunately, KGs 
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are typically manually curated meaning that integrating new information 
typically requires several days. Misinformation, however, has the potential 
to cause significant damage in just a few hours. As a result, such solutions 
relying on fact checking sites such as Snopes or Politifact are not suitable for 
early detection algorithms.7! 

The U.S. National Security Commission on Artificial Intelligence noted 
that AI could be used to manipulate training data with the potential for 
compromising Al learning.”” There are a few approaches for detecting mis- 
information. Unfortunately, it is difficult to judge which is the most effective 
as there is no standard benchmark dataset that might be used to make a fair 
comparison.” These approaches are often complex and are beyond the scope 
of this chapter. We will briefly outline them here. 

The content-based approach uses content of the information rather than 
considering how that information is propagated. For example, a click bait- 
like format of a headline or sentiment polarity might be an effective dis- 
criminator. This approach has advantages as it can be used for early detection 
algorithms.2* The social-response-based approach investigates how users re- 
act to a post. Responses such as “fake” or “untrue” indicate that users perceive 
misinformation. Emotional responses such as “unbelievable!” and political 
term such as “PC shit!”?° also correlate with inaccurate information.”° 

A hybrid approach combines the social response of users with their charac- 
teristics. Each user is given a suspiciousness score based on their interaction with 
the system. The suspiciousness score of a piece of news is then simply a sum of 
the corresponding scores of the users interacting with the post. This approach 
was first used in Capture, Score, Integrate (CSI). Similar approaches were im- 
plemented in dEFEND and FNED.”’ The graph-based approach investigates 
how information spreads across a network. The underlying hypothesis is that 
inaccurate information spreads differently than accurate information and can 
be identified by investigating their pathways and the way users respond and 
interact with them. TraceMiner and FANG are two examples of successful 
algorithms using such techniques.” Finally, the multimodal approach in- 
corporates information from multiple modalities to build better predictive 
models. The simplest example is to combine visual and textual information. 
This was shown to boost baseline accuracy by up to 7%. 

We continue to learn how social networks evolve and how users interact 
with each other, spreading misinformation and learning from each other. 
Tools are improving for understanding such networks. Since these processes 
are complex and interact with each other, often the only way to see the 
outcome of a given action is by performing extensive simulations. Through 
combining all of these techniques, we can try to detect malicious behaviour, 
understand the impact it has, or even strike back and counterbalance such 
actions. 

Understanding the processes shaping social networks is an important task 
for social scientists trying to understand human behaviour but also for data 
scientists designing algorithms and tools working on those networks. Better 
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knowledge and dedicated tools may allow predicting which posts will be 
popular, which users have the power to change the shape of the network, 
which political view will likely become dominant, and, more importantly, 
how to detect if someone affects this processes and how to counterbalance 
their effects. 

In the area of social network dynamics, there are several aspects that are 
relevant for understanding potential threats: 


e How links between social agents are formed. What structures of links 
emerge naturally, and which structures are most likely to be engineered 
by malicious actors? 

e How misinformation spreads in a network. Can the speed of its spread be 
predicted and influenced including identifying the social agents that are 
most likely to help spread misinformation. 

e How access to information, sources, timing, dynamics, and form, influ- 
ences the opinions of social agents. Specifically, how malicious agents 
can shape the listed factors to maximize the effectiveness of adversarial 
campaigns. 


One of the first models of a dynamic network is the preferential attachment 
model introduced by Barabâsi and Albert,“ who observed a power law degree 
sequence for a subgraph of the World Wide Web soon after the property 
was observed for the internet graph.*! This property conforms to the Pareto 
principle, which posits that about 80% of effects result from 20% of causes. 
For example, the top 20% of earners in United States pay roughly 80-90% 
of Federal income taxes. The preferential attachment model explains why 
power-law degree distribution, displaying a high degree of asymmetry, oc- 
curs in many real-world complex networks such as the rich-get-richer phenom- 
enon. The preferential attachment rule incorporated in the model makes new 
nodes to be more likely to connect to the more connected nodes than to the 
smaller nodes, creating a power-law degree distribution. 

There are many other important models explaining various properties such 
as increasing edge density (densification) and decreasing diameter or surpris- 
ingly large clustering coefficient and short average path lengths, the proper- 
ties explained by the Watts-Strogatz model.*? Below we concentrate on a few 
aspects that need identification. 

For detecting malicious activity, the key aspects of social network anal- 
ysis are: 


e Certain local structures of connections between nodes in the social graph 
that can be considered as suspicious, often referred to as motifs. 

e Overly regular sub-graphs of a social network. Empirical analyses show 
that social ties are characterized by the presence of certain levels of noise. 

e Anomalous nodes with vastly different characteristics in comparison to 
what would be predicted for them. 
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Social networks create an information platform in which human and software- 
assisted accounts (bots) try to take advantage of the system for various reasons 
including triggering collective attention,’ gaining status," monetizing pub- 
lic attention,” spreading disinformation, or seeding discord.’ 

A large number of Twitter accounts are bots that are responsible for much 
of the disinformation shared. Recent studies showed that bots play an im- 
portant role in the initial stage of diffusion by amplifying low-credibility 
content, but they cannot distinguish between true and false information. 
Humans, however, are more likely to spread false information.” It is pos- 
sible to characterize the behaviour of individuals who use bots to enhance 
their online visibility and influence. Such accounts, bot-assisted humans or 
human-assisted bots, are often referred to as cyborgs or augmented humans.°® 
Opportunities for spreading (mis-, dis-)information and banning strategies 
depend highly on whether a social platform is moderated or not. Finally, 
various countries exhibit different levels of infodemic risk, adding yet an- 
other level of complexity.*° 

Understanding of social networks has advanced significantly, but the 
underlying mechanisms for spreading false information are still not well 
understood. Recent results suggest that spreading mechanisms might have 
indistinguishable population-level dynamics.*! We are flooded with infor- 
mation at a level impossible to consume. Human attention is limited and 
individual reaction to information is a complex interplay between individual 
interests and social interaction. Collective attention is typically characterized 
by a quickly growing focus on a specific topic, such as presidential elections 
or vaccination, until a well-identified peak of attention is reached. This is 
followed by the second phase in which a slow decay of interest is observed.” 

Initial research neglected the effects of the underlying social structure but 
it is clear that underlying network structure plays an important role in the 
process.” De Domenico and Altmann combine two simple mechanisms to 
explain the dynamic of collective attention: a preferential attachment process 
shaping the network topology and a preferential attention process responsible 
for individual’s attention bias towards specific users of the network. 

Social learning refers broadly to the processes by which a person’s social en- 
vironment shapes how a person behaves and thinks. In the context of social 
networks, a user may adopt the cognitions or behaviours from those they 
interact with directly. Concurrently, learning also shapes the social environ- 
ment, since individuals also exercise control over their social environment 
and potentially select network partners as a function of individual attributes 
including behaviours and cognitions.” As a result, social learning is a complex 
process but important in understanding the spread of misinformation. To il- 
lustrate its complexity, we focus on two aspects: segregation and polarization. 

Various models explain why and how segregation occurs beginning with 
the classic theory of residential segregation.” Segregation theory also applies 
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to the structure of networks; segregated networks always emerge even if the 
users are assumed to have only a small aversion from being connected to 
others who are dissimilar to themselves and yet no actor strictly prefers a seg- 
regated network." The power of aversion is often amplified by homophily, 
a tendency for people to have ties with similar others." For example, Bener 
et al. developed and tested empirical models of how social networks evolve 
over time; particularly, how people in a social network choose links on the 
basis of their own attributes, and how individual attributes are in turn shaped 
by network structure.” 

An extreme situation, polarization, occurs when a group is divided into 
two opposing sub-groups having conflicting and contrasting positions, goals, 
and points of view with few individuals remaining neutral. Polarization typ- 
ically occurs in politics but several other issues often experience it including 
climate change, gun control, same-sex marriage, and abortion. The presence 
of polarization changes the dynamics of a network. Indeed, for example it 
is known that Twitter users are unlikely to be exposed to cross-ideological 
content from the clusters of users they followed, as these groups were usually 
politically homogeneous.”” Antonakaki, Fragopoulou and Ioannidis identi- 
fied this in a recent survey of Twitter research.”! 


Hypergraphs 


Most network science concentrates on building tools for mining complex 
networks represented as a graph. Facebook, for example, might be repre- 
sented as a set of nodes associated with users and a set of edges between nodes 
associated with friendship relationship between the users. Most of the social 
media platforms, especially forums, are more complex and require more so- 
phisticated representations to properly represent them. Interactions between 
users commenting on a given post may be represented as a tree rooted at the 
node associated with the user who posted the initial post and internal nodes 
in the tree associated with users who comment on someone else’s comment. 

Current technologies and tools are not capable of dealing with such com- 
plex structures but there is movement to the next level introducing tools 
that deal with hypergraphs. In this simpler structure, nodes form hyperedges 
that may consist of any number of members, not only two as in the case of 
graphs. A hyperedge may include all users that interacted with a specific post, 
or “liked” a particular photo on Instagram. There has been a recent surge 
of interest in higher-order methods, especially in the context of hypergraph 
clustering.’ Battiston et al. provided a recent study on the higher-order ar- 
chitecture of real complex systems,°? and for a hypergraph neural networks 
framework (HGNN) for data representation learning.” 

Another recent effort has been to represent each node of a network as a 
low dimensional feature vector. There are a number of reasons for using this 
approach. On the one hand, it decreases the dimension of the problem mak- 
ing it easier to handle and analyse. Moreover, even though dimensionality 
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reduction means losing some information, it is actually a useful technique. 
Indeed, well-trained algorithms aim to learn the most important informa- 
tion about the network but ignore “noise”. As a result, “compressed” rep- 
resentation is often more useful and informative than the complete picture 
(an analogue situation would be as follows: reading a relatively short sum- 
mary of the two-hours long meeting might be more useful than listening to 
the original recording). As a result, over 100 node-embedding algorithms 
have been proposed recently in the literature. The techniques and possible 
approaches to construct the desired embedding can be broadly divided into 
the following families: linear algebra algorithms, random walk-based meth- 
ods, and deep learning methods.” Selecting an appropriate embedding for a 
given network and a given task at hand typically require ad-hoc experiments 
and tests performed by domain experts. However, a general framework was 
recently proposed that provides a tool for an unsupervised graph embedding 
comparison, which should make the selection process easier, and the outcome 
of better quality.°° 

The network dynamics and information spread in them is complex. We 
have highlighted a few aspects, but this is only a starting point. This com- 
plexity is driven by several factors. We are faced with a very large number of 
agents in the system. Each has its own characteristics making them distinct. 
In the literature this feature is called agent heterogeneity.>” The behavioural 
rules of agents are typically complex, depend on their characteristics, and 
evolve over time. The description of the microstructure of the social network 
is quite extensive. In most cases, we are interested in some macro conse- 
quences of this microstructure such as the speed and reach of malicious infor- 
mation. Experience in modelling of enormous systems show that even small 
changes in microstructure can lead to complex and non-obvious changes 
in macro results. In the literature, such consequences are called emergent, 
as they are frequently impossible to deduce given only the microstructure 
assumptions. 

A typical approach to address these challenges is to construct a digital twin, 
a synthetic model of a real system.” The representation of the real social net- 
work in such an approach is usually done in 1:1 scale. The calibration of the 
microstructure of the system is made based on available individual and ag- 
gregated data although it is sometimes impossible to collect all relevant char- 
acteristics of the system on an individual agent level. Such a representation is 
done using software implementation.” Given the scale of the simulations the 
crucial requirements are high-performance tools used to create such models. 

Virtual simulators of the social network interactions allow for performing 
flexible what-if analysis that provides verification of how the system would 
react to different stimuli. It is important to stress here that very often the 
modelled systems are highly volatile and the response of the system is hard 
to predict. Therefore, the simulations usually are run multiple times creating 
a distribution of potential outcomes rather than a single point estimate.°! 
Such approaches are commonly used to find optimal seeding of social media 
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campaigns and find the agents in the social network who should be used to 
initialize spreading of information to ensure its maximum reach and speed.°* 

The simulation approach is often coupled with metamodeling, also called sur- 
rogate modelling.°° The metamodel of the simulation provides a simplified view 
and typically uses supervised learning techniques and Explainable AI (XATI) 
tools.0* The objective of creating of a metamodel is twofold. First, analysts 
working with a complex simulation model are usually interested in which 
components determine the direction and impact of the outcomes. The goal 
of the metamodel is to provide a simplified view of the complex system in 
which accuracy is traded for easier interpretation and greater understanding. 
Such qualitative insights are often invaluable for understanding the grand 
principles that control and influence social networks can apply. The second 
use of the metamodels is related to speed with which they can produce pre- 
dictions. Often the original simulation model can take hours or days to pro- 
duce outcomes. In contrast, metamodels can produce results within seconds, 
albeit with less precision, a trade-off between speed and accuracy. Such a 
trade-off is often desirable in applications where alternative courses of action 
need to be evaluated and action taken quickly. 


Conflict among the major competitors 


COVID-19 has been used to advance nation-state political agendas and sup- 
port their efforts in competition and conflict. The health emergency has 
enabled campaigns bearing the characteristics of hybrid warfare with the in- 
telligence services of both the European Union and United States recognizing 
the intensified activity of foreign actors.°° The use of asymmetric methods 
or hybrid warfare techniques and procedures should be viewed as activities 
along a spectrum of conflict short of kinetic warfare. These methods attack 
both the target state’s population as well as attack the decision-making process 
of the target state.” The concept of hybrid warfare has been used to refer to 
the combination of military tactics such as conventional warfare with irregu- 
lar warfare and cyber warfare, as well as the employment of other subversive 
instruments and tactics, to achieve two goals: first, to avoid responsibility 
and retribution, and second to weaken and destabilize the enemy without 
perceived involvement.°* Hybrid warfare influence campaigns include prop- 
aganda, malinformation, and disinformation to influence the thoughts and 
beliefs of the target population.” A defining element of hybrid warfare is 
targeting of the population alongside traditional political, economic, and in- 
formation aspects of conflict. The pandemic crisis has been used to spread 
misinformation and disinformation to further political objectives and expand 
national influence.’! Early in the crisis, both Russia and China have used 
the pandemic to further their political goals such as sending medical advisors 
and supplies to Italy. 7? The Pew Research Center found that prominent of- 
ficials and news media outlets in Russia, China, Iran, and the United States 
became super-spreaders of disinformation about COVID-19.75 Russian and 
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Chinese media efforts consistently seek to undermine confidence in Western 
developed vaccines including reporting false information including sensa- 
tionalizing safety concerns and promoting the narrative that their vaccines 
are superior to those produced in the West.7* A significant amount of misin- 
formation related to the pandemic has come from credible sources increasing 
the problem of distinguishing accurate from inaccurate information.” AI 
includes powerful tools to both generate misinformation and detect its use. 
As Kasapoglu and Kirdemir note, Al could cause drastic changes in hybrid 
warfare, which is a major concern for NATO.”° States and non-state actors 
can use cyberspace to influence large groups of civilians and opposing forces. 
From reconnaissance activities and the profiling of target audiences to the 
weaponization of distorted or fake information and psychological operations, 
AI broadens the potential of information operations. The problem being con- 
sidered is how to combat misinformation in cyber space using AI to identify 
sources, discover how misinformation travels, and counter its adverse effects. 
Concurrently, AI can be weaponized in information operations making it 
both a weapon and a threat. 

Infodemics is a term first used to describe the information overflow dur- 
ing the 2003 Severe Acute Respiratory Syndrome (SARS) epidemic.” In 
the beginning of the COVID-19 pandemic, the World Health Organization 
(WHO) noted that the health crisis was accompanied and amplified by the 
astonishing data overload and unprecedented information chaos, developing 
at the extraordinary pace, and began to use the term of Infodemics to address 
it. 8 The WHO also called to unify the vocabulary related to Infodemics, en- 
dorsing the creation ofa unified lexicon allowing for the clear understanding 
and use of terminology among the community of experts.” Such approach 
was supported by the subject matter experts®? and the following vocabulary, 
adapted from Wardle and Derakhshan,*! has been in use among the info- 
demic management community since. 

Disinformation is false information that is deliberately created or dissem- 
inated with the express purpose to cause harm; fabricated content and the 
malicious intent behind it are the identifying properties of disinformation. 
The term misinformation is frequently and erroneously used as a synonym of 
disinformation, but, although the disseminated information is false, it is not 
intended to cause harm, as the persons sharing it believe that it is true and 
accurate and attempting to be helpful. Finally, malinformation is genuine in- 
formation that is broadcasted with a deliberate intent to cause harm, by re- 
vealing data that may hurt the reputation of a person or an institution.** It is 
important to highlight, that infodemiology experts reject the term fake news, 
which was coined to describe the use of disinformation and misinformation 
in news reporting.5* The term has been used by political actors to discredit 
news reporting and reported facts they dislike.5* 

The COVID-19 pandemic has highlighted conflict in cyberspace among 
three major competitors: Russia, China, and the United States. Russia 
and its predecessor, the Soviet Union, have had a longstanding adversarial 
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relationship with the United States and its Western allies. China, in its quest 
to become the predominant state power, has utilized its cyber capabilities 
to advance all of its instruments of national power. In the United States, 
many aspects of the pandemic have become politicized, often used as weap- 
ons between the major political parties over measures to contain the virus and 
vaccinations against it. In the process, the United States has become a major 
source of misinformation that affects much of the world. Each of the major 
powers makes significant contributions to conflict in cyber space in its own 
way and for different reasons. 

Russia has maintained a prolonged effort to undermine the legitimacy 
of Western democracy and democratic societies.5 Putin perceives that the 
greatest threat to Russia is Western democracy and attacks the West to shift 
attention away from weaknesses including corruption and the economy.5€ 
The 2012 plan to increase economic growth was poorly conceived and lacked 
follow through.“ The gross domestic product (GDP) for 2020 is estimated 
to be about $1.5 trillion, 11th in the world; however, per capita GDP was just 
$11,654 or 64th of all countries.5% Russian economic and political power is 
concentrated in the hands of a small number of individuals made up of former 
intelligence officers and oligarchs.*? 

Russia regularly conducts covert action including spreading misinforma- 
tion to exploit societal divisions and undermine democratic legitimacy.”? 
Russian use of a health crisis to further its political agenda is not new. During 
the 1980s, Soviet intelligence disseminated disinformation that HIV/AIDS 
virus was developed by the United States as a biological weapon.”! Russia 
has echoed Chinese claims that COVID-19 was developed and spread by the 
United States as a weapon against China.”” Russian trolls have amplified dis- 
information to exacerbate the American debate on vaccines and undermine 
American politics. 

Russian Federation disinformation is a descendant of USSR’s political war- 
fare theories and practices.”* Soviet KGB operations included subversion, me- 
dia manipulation, propaganda, political repression, political assassination, the 
establishment of opposition parties and criminal organizations.” They are 
similar to the current practices of the Russian Federation’s Federal Security 
Service (FSB) using political assassination, political tyranny at home, and 
sponsorship of paramilitary groups in independent nations. Their methods 
also include the use of soft power such as worldwide disinformation cam- 
paigns, foreign political meddling, and the establishment of networks of in- 
fluence abroad.” 

Putin’s strategies use a combination of repression at home, disinforma- 
tion and destabilization campaigns in the West, and military interventions in 
Ukraine and Syria. All result from the simultaneous strength and insecurity 
of the Russian leader, who struggles to manage a corrupt, kleptocratic, mafia 
style state.” Perhaps the Kremlin's policies are not a part of a grand strategy, 
but an opportunistic foreign policy. Putin probes for Western weakness, ir- 
resolution, and indecision, and then, if there is no resistance, he intervenes 
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to extend Russia’s reach by absorbing, either physically, or by exerting influ- 
ence, more territory. S Russia has begun to devote extraordinary resources 
to a political and information war designed to undermine and eventually 
destroy Western democracies and institutions.” Moscow is pursuing invest- 
ments in high-impact, low-cost asymmetric warfare to correct the imbalance 
between Russia and the West in the conventional domain.!”° 

China aspires to be the leading economic, military, and political player 
in the world, seeking to surpass the United States and its Western allies in 
all aspects of national power by 2035.!°' The U.S. Department of State has 
identified China as the central threat that undermines the stability of the 
world, identifying issues including predatory economic practices, disregard 
for human rights, and undermining global norms and values.1%2 Made in 
China 2025 is an ambitious plan to upgrade Chinese industries, especially in 
advanced fields including robotics, AI, and quantum computing. The aggres- 
sive nature of the plan may motivate the use of questionable trade practices 
including cyber theft.1% To reach its 2035 goals, China must continue to 
raise its industrial productivity through capital inputs, human resources im- 
provements, and acquiring advanced technologies.'°* China’s ability to reach 
this aggressive goal is in question.!% The GDP for 2020 is estimated to be 
about $15 trillion, 2nd in the world; however, per capita was just $11,819 or 
61st of all countries.!°° This may lead to even more aggressive efforts to ac- 
quire advanced technologies. 

China uses a network of related media organizations to propagate misin- 
formation online and harass opponents at home and abroad. It produces and 
echoes misinformation and disinformation on issues including election voter 
fraud, COVID-19, and QAnon theories.'°? China is modernizing all aspects 
of the People’s Liberation Army from strategic nuclear forces and cyber ca- 
pabilities to conventional capabilities of all types.'°* China continues to con- 
duct espionage against the United States on a massive scale. They have been 
successful in stealing designs for advanced weapons systems including the 
F-22 and F-35 fighter jets and C-17 transport aircraft.'"? Recently, Chinese 
fighter jets have surpassed Russian aircraft performance, in part, due to their 
willingness to conduct industrial espionage and reverse engineer advanced 
technologies.!!° Chinese pharmaceutical companies are behind their Western 
competitors on mRNA technology, used to rapidly produce COVID-19 vac- 
cines.!!! Two Chinese government-linked hackers attempted in 2020 to steal 
data related to mRNA technology from pharmaceutical firm Moderna.!!” 

In the United States’ politically polarized society, news outlets, fringe me- 
dia and conspiracy sites — some with significant global reach — mislead their 
audiences with false narratives with significant negative outcomes.'!? Ac- 
cording to a U.S. national survey during the early days of the COVID-19 
pandemic, the respondents’ identified political party, correlated with spe- 
cific beliefs about protection from infection. Conservative media use, how- 
ever, correlated with belief in conspiracy theories." Additionally, a 2021 
survey by the U.S. National Academy of Sciences found that 75% of persons 
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overestimated their ability to identify misinformation and the most overcon- 
fident were poorest at it and most likely to spread the information. 115 

The U.S. information chaos related to the COVID-19 pandemic was am- 
plified by propagation of conspiracy theories.'!° Theories ranged from link- 
ing the disease with the 5G communications through QAnon community 
claims that the virus was created the government officials and other fig- 
ures that secretly run the world. Such theories generated political and social 
effects, and increased anxiety, uncertainty, distrust towards authorities and 
fear.''? Conspiracy theories in the QAnon community have posited that the 
coronavirus might not be real and that if it was, it had been created the gov- 
ernment officials and other figures who secretly run the world. 

COVID-19 misinformation in the United States came from both offi- 
cial and unofficial sources with devastating effects there and potentially 
worldwide. Evanega et al. found that about one third of English language 
COVID-19 misinformation were news media mentions of statements by U.S. 
President Donald Trump, of which just 16% was fact checked.!!* The rest of 
the media content was provided without question or confirmation. Brennen 
et al. observed that prominent public figures including politicians and celeb- 
rities accounted for 20% of pandemic misinformation, however, these indi- 
viduals accounted for 69% of the misinformation echoed on social media.!!” 
The U.S.-based Center for Countering Digital Hate found that just 12 people 
in the U.S. were responsible for about two thirds of misinformation support- 
ing vaccine hoaxes and vaccine hesitancy. These persons include anti-vaccine 
activists, doctors, and alternative health promoters whose organization gen- 
erate up to $1 billion annual from these activities.” Belief in misinformation 
related to COVID-19 is associated with vaccine hesitancy and noncompli- 
ance with health guidance.'”! 

In 2018, Google allow a contract with the U.S. Department of Defense 
to lapse because of employee opposition. !?? Operation Maven was intended 
to use AI with drone surveillance video to identify targets and support op- 
erations against them.!?° In March 2021, the U.S. National Security Com- 
mission on Artificial Intelligence identified potential uses of AI in conflict 
including AI enabled information operations. These threats include auton- 
omous disinformation campaigns and computational propaganda including 
deepfakes.!74 There has been, however, some backlash against the use of AI 
in warfare. 


Conclusions 


Considered together, modelling can help to understand how links between 
agents are formed, how information is disseminated in a network, and how 
this information, combined with other factors, can influence opinions and 
actions of agents active in social networks. Together they have significant po- 
tential to blunt the effects of misinformation related to the pandemic as well 
as other types of misinformation campaigns such as anti-vaccination efforts. 
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Malicious agents can use the tools, usually with the goal of inciting some 
behaviour by people engaged in social network. Therefore, they use the mod- 
elling of the system to plan their actions to maximize impact. Typical actions 
in this area would involve injecting bots into a network, hijacking neutral 
nodes, or interfering in the communication between nodes. All the actions 
are typically aimed at distorting the structure and information flow within a 
network. The strategies used here are usually staged, during an initial phase 
the malicious nodes would act “normally” and, only after gaining credibility, 
start their detrimental activities. 

These same tools can be used to fight malicious agents. The tools and tech- 
niques discussed can be used to detect anomalous activities in networks or 
network structures. When adversarial information is identified using either 
human analysis or analytical tools, the simulation and social network analysis 
techniques can be employed to identify sources of malicious actions within 
the network. Identification of the sources of misinformation may blunt the 
effectiveness of their messages and erode the perception of the veracity of 
what they disseminate. 

Use of these AI tools has the potential to identify sources of misinformation, 
study how the misinformation moves through networks, and ultimately blunt 
the effects of the misinformation. We recommend testing these methodolo- 
gies against the misinformation surrounding COVID-19 and the vaccinations 
against the virus. Several the sources of misinformation on both are known, al- 
lowing for validation of the AI techniques and training AI tools while support- 
ing combating the misinformation and disinformation related to the pandemic. 
This effort will likely uncover additional false narratives as well as identify addi- 
tional malicious actors. It will also increase acceptance of AI methodologies by 
nation-states, civil society, and individual consumers of information.” 
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4 Algorithmic power? 


The role of artificial intelligence 
in European strategic autonomy 


Simona R. Soare 


Introduction 


The European Union and its member states are investing in a wide range 
of emerging and disruptive technologies! (EDTs) to be the backbone of 
European strategic autonomy. The Sino-American strategic and technolog- 
ical competition is pushing Europe into a geopolitically tight space. Brus- 
sels fears the geopolitical and geoeconomic consequences of lagging behind 
the US and China, France fears “irreversible dependencies”? and Germany 
fears the loss of European geoeconomic competitiveness? if Europeans do not 
strengthen their technological sovereignty and strategic autonomy. 

The US wants Europe to invest more in defence and safeguard against 
malign Chinese investments in key technologies and critical infrastructure. 
However, Washington remains sceptical of European strategic autonomy for 
geopolitical reasons, particularly if it leads to a European “third way” or 
European “equidistance” between the US and China.* Beijing is encouraging 
European strategic autonomy notably from the US, and Russia is sceptically 
weighing the alternative futures of the European project.° In short, interna- 
tional actors do not see sufficient strategic intent and technological and hard 
power behind the EU’s narrative on strategic autonomy. As demonstrated by 
the 5G debate, the US and China perceive Europe as the grounds for geopo- 
litical confrontation — an outcome Brussels and major European capitals badly 
want to avoid. 

Investing in European-grown and controlled EDTs (including support- 
ing data and digital infrastructure) and defence capabilities has become a 
European priority linked to strategic autonomy. And the COVID-19 pan- 
demic has accelerated European strategic planning in this regard. Technology 
is key to both self-perceptions and international perceptions of the EU status 
as one of the leading geopolitical players.° Strategic autonomy requires a de- 
gree of differentiation from and sustainable competitiveness with other great 
powers, notably the US and China.’ Arguably, this is the impetus behind 
what European Commission president von der Leyen calls “a digital transi- 
tion which is European by design and nature.” 

What role do AI technologies play in European strategic autonomy in 
security and defence? At present, the EU is ill-equipped to leverage AI 
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technologies in defence and transfer them to the battlefield in ways that 
lead to greater strategic autonomy in defence. The paper argues insofar as 
European strategic autonomy is concerned, the adoption of AI technologies 
in defence is a distraction because it introduces new layers of complexity 
to European defence without significantly contributing to greater European 
strategic autonomy. 

Four interrelated reasons explain this. First, Europeans lack a common AI 
integration strategy in defence which links technological power to strategic au- 
tonomy in terms of operational advantage against and competitiveness with 
other rival great powers. The EU does not have, nor does it currently plan 
to develop a common European military strategy to integrate AI in cyber and cross- 
domain military operations for operational and strategic advantage and it does not 
possess a common, regular threat and opportunity assessment based on European intel- 
ligence about its rivals’ AI military innovation efforts and other international 
actors’ geopolitical needs. The EU is not politically ready — or interested — to 
develop the kind of military capabilities, enablers, and legal powers to con- 
duct algorithmic warfare. 

Second, Europeans are still struggling to close important technology and capability 
gaps, including in the fields of cyber and digital emerging technologies. How- 
ever, as cyber, and digital technologies become the backbone of cutting-edge 
military capabilities and shift the global balance of power, greater strategic 
autonomy depends on European mastery of these new forms of power as 
effective tools of military statecraft. So far, the EU’s approach to strategic 
autonomy and technology lacks a clear geopolitical and proactive focus, and 
resides firmly in a normative, regulatory, and reactive dimension that is de- 
void of a strong technological and hard power foundation. 

Third, EU decision-making in foreign, security and defence policy is ill-equipped 
to accommodate the use of AI technologies. There is currently no clear under- 
standing of what role AI would play in supporting EU decision-making and 
of the differentiated information needs from AI systems at different levels of 
decision-making, across the civil-military spectrum. Ongoing debates about 
Qualified Majority Voting (QMV) and article 44 TEU do not offer an ade- 
quate solution because they do not reflect deeper adaptations needed to ac- 
commodate and integrate the use of AI. Insofar as AI can deliver information 
and not political consensus, it is arguable whether AI technologies are an ad- 
equate cure for Europe’s ‘bias for inaction’ which stems from a fundamental 
lack of political will to act, common threat assessment and agreement on how 
to tackle these threats, including through the use of force. 

Fourth, the role of AI technologies in enabling greater European strategic auton- 
omy is secondary to geopolitical and exogenous factors. Geopolitical and structural 
factors such as great power competition and threat perception, the structure 
and functioning of the global internet, global AI standards, strategic supply 
chains, European relations to private industry, interdependence and global 
geopolitical alignment patterns are powerful constraints that affect European 
strategic autonomy regardless of the availability of European technologies. 
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This paper seeks to break new ground by providing a conceptual and 
empirical analysis of the relationship between technology and political au- 
tonomy in international organisations, alliances, and multilateral fora. It 
contributes to security and technology literature by conceptualising the re- 
lationship between AI technologies, the shift in the distribution of global 
power and political constructs of power and autonomy. It does so by framing 
European strategic autonomy as a strategy for great power competition at four 
levels: decision-making, technological, operational autonomy, and partner- 
ships. The paper also highlights the pitfalls of techno-centric solutions and 
debunks frequent assumptions about the equivalency between mastering new 
technologies and employing them as effective tools of statecraft in strategic 
competitions. 

The paper proceeds in three sections. Section one explores the link be- 
tween European strategic autonomy, Al, and cyber power. Section two of- 
fers a radiography of the EU’s efforts and strategy development towards AI 
adoption and assesses their early results. The third section analyses European 
efforts to adopt AI in defence in relation to European decision-making au- 
tonomy, technological autonomy, operational autonomy, and defence part- 
nerships, notably EU-NATO cooperation. 


Algorithmic power and strategic autonomy 


A strategy for international competition? 


Since 2016, the key EU objective has been to achieve a generically defined 
“appropriate level of ambition and strategic autonomy”® “while preserving 
an open economy.” The European Council defined strategic autonomy as 
the “capacity to act autonomously when and where necessary and with part- 
ners wherever possible.”!” The concept encompasses four main dimensions: 
decision-making autonomy, technological autonomy, operational auton- 
omy,|! and partnerships. 

European strategic autonomy has developed and has been used in a 
European defence and defence industrial context,” in relation to the 
European development, possession, projection, and employment of hard 
power. The COVID-19 pandemic and the rapid changes in the international 
system!? contributed to the broadening and refinement of the meaning of 
strategic autonomy as well as to its prioritisation." President of the Council, 
Charles Michel reflected this trend by defining strategic autonomy as “more 
resilience, more influence. And less dependence” on other international ac- 
tors. Strategic autonomy is about “being able to make choices” by “reducing 
our dependencies, to better defend our interests and our values.” "5 

According to this broader understanding, strategic autonomy is the EU’s 
long-term strategy for international great power competition. It is driven 
by the recognition of a shifting global balance of power, the develop- 
ment of autonomous decision-making capacities and capabilities (political, 
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institutional, and material; soft and hard power), the reduction of critical ex- 
ternal dependencies, and the emulation!” of other great powers geopolitical 
behaviour. This is further confirmed by the apparent conceptual overlap with 
another concept the EU and European states use in parallel, namely “strategic 
sovereignty.” !® 

Both strategic autonomy and strategic sovereignty encompass internal and 
external dimensions. They concern: deepening integration (possibly towards 
an implied federalist project denoted by the use of the concept “sovereignty”) 
and the full use of EU treaty provisions; domestic resilience and the reduction 
of strategic dependencies on external actors, securing critical supply chains 
and controlling critical infrastructure; the EU’s internal capacity to com- 
mand and control critical technologies which are key to its economic inter- 
ests and security; and the EU’s capabilities and political will to act externally 
to defend its strategic interests.” Both concepts seek to empower the EU and 
the member states acting collectively,” as exemplified by the expressed desire 
for more flexibility between CSDP and ad hoc missions of the member states 
in the Strategic Compass negotiations.” They also signal an expectation of 
international recognition by other great powers of the EU and the Europeans’ 
peer status as credible security providers. 


Weak links 


Nevertheless, European military strategic autonomy lacks credibility.” 
Europeans lack credible military capabilities and strategic enablers?” in suffi- 
cient numbers and levels of readiness, interoperability, and digitalisation for 
modern warfare.2* CSDP lacks a credible military level of ambition”? as do 
different European minilateral cooperation formats. The Helsinki Headline 
Goals for a corps-level EU Rapid Reaction Force, deployable in 60 days and 
fully sustainable in the battlefield for up to a year, were never met and, in 
practice the level of ambition has been much lower.”° The EU Battlegroups 
were created, but they were never operationally deployed. 

Furthermore, digital technologies in and beyond the military domain 
represent an EU strategic vulnerability. While Europeans have been mak- 
ing progress in Al, this progress varies widely across AI technology stacks. 
Europe’s “share of global patent applications in Big Data is the smallest among 
all Advanced Technologies,” no European chip manufacturer produces com- 
petitive, cutting-edge semiconductors?” for advanced AI and QC (Quantum 
Computing), and “the largest European cloud service provider accounts for 
less than 1% of total revenues generated in the European market.”?> Between 
2009 and 2019, leading European states such as France and Germany relied 
73% and 64% respectively on national investors for the funding of tech start- 
ups. The rest of the funding came mainly from the US and the UK.” 

Since the late 2000s, European strategic dependency on rare earths and 
raw materials necessary for digital technologies and defence capabilities has 
increased. For many rare earths elements such as antinomy, which are used in 
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electronic-optical systems and semiconductors, Europe relies on Chinese sup- 
pliers and faces critical supply chain choke points.” Europe equally depends 
on limited foreign suppliers for digital components and semiconductors.*! 

Both security and defence, and digital technologies remain weak links in 
the EU’s quest for strategic autonomy and status recognition as a geopolitical 
actor. 


Creating European AI leadership 


Progressive refinement of AI goals 


Building on the priorities of the EU Strategy on Artificial Intelligence 
(2018),°7 the 2021 review of the Coordinated Plan on Artificial Intelligence marks 
a higher EU AI level of ambition by asserting the EU’s qualitative AI leader- 
ship. Furthermore, the 2020 White Paper on Artificial intelligence focuses on 
enhancing European AI cooperation through cross-border networks of excel- 
lence and pipelines of innovation as a means of reducing the significant Al 
policy, technological capacity, and output fragmentation between European 
states. The document seeks to achieve this by providing options for regula- 
tory frameworks to support AI adoption and/or mitigate against legal and 
ethical risks from the use of these technologies. 

Finally, the 2021 Artificial Intelligence Act is the EU’s effort to shape the 
(global) standards of AI and assert its AI qualitative leadership. Following 
in the footsteps of the Global Data Protection Regulation (GDPR), this is 
perceived by the Commission as the materialisation of the EU’s first mover 
advantage to establish a balanced approach between sufficient regulation and 
facilitating innovation, between security and civil liberties. 


Cross-policy synergies 


The EU seeks to create synergies and correlations between different in- 
dustrial sectors, including incentivise spin-off and spin-in** effects by es- 
tablishing cooperative frameworks (e.g., defence innovation networks, and 
innovation accelerators and incubators), promoting hybrid civil-defence 
technology standards, reducing duplication of effort and funding, and better 
coordinating European-based R&D&I and procurement, through the Digital 
Compass (2021)%5 and the Action Plan on Synergies between the civil, defence and 
space industries (2021). The Action Plan proposes three steps: first, to identify 
critical technologies and create an EU Observatory of Critical Technologies 
to “provide regular monitoring and analysis of critical technologies, their po- 
tential applications, value chains, needed research and testing infrastructure, 
desired level of EU control over them, and existing gaps and dependencies.” 
Second, based on these classified reports, to develop and implement technol- 
ogy roadmaps that serve as the basis for concrete EU actions, capabilities, and 
decision-making processes that enhance its overall technological sovereignty. 
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And third, to pursue flagship projects that maximise European and cross- 
sector technological cooperation. 

There are indications that EU digital policies have matured over the past 
five years. The Union has established and updated policies in all critical sec- 
tors for AI adoption, including semiconductors, cyber space, data and digital 
networks and infrastructure, including 5G and cloud. The 2020 EU Cyber- 
security Policy acknowledges the role of AI technologies in cyber defence and 
deterrence across the civilian and defence domains as well as the role cyberse- 
curity plays in securing Al, the Internet of Things, and other emerging tech- 
nologies.” In 2021 the European Union Agency for Cybersecurity (ENISA) 
published an AI threat landscape study outlining the main threat vectors and 
threat actors and proposed the establishment of “an AI toolbox (...) with con- 
crete mitigation measures” for the threats to European AI systems.” How- 
ever, the EU is hard pressed to find members that would contribute offensive 
cyber capabilities to its CSDP missions or to defend EU AI models. 


A broad public mandate 


The focus and acceleration of EU AI efforts are much needed in view of 
European strategic vulnerability and high public “digital distrust” and 
techno-nationalist tendencies in leading European countries. European 
publics and expert communities reflect these techno-nationalist tendencies: 
“trust in digital technology does not extend far beyond national borders: 
Europeans are sceptical of governments and companies from other European 
countries. This “digital distrust” is only exacerbated when looking across the 
Atlantic” and vis-à-vis China.*? Recent surveys suggest a growing European 
public concern with the societal and security impact of AI, automation, 
and other modern weapon systems" and national dependencies on Chinese 
(54%), American (50%), and other EU (42%) digital technologies. 


Early results of European AI efforts 


As a result of Brussels’ market structuring efforts, by July 2021, 20 EU mem- 
ber states and all four EFTA associated countries had adopted AI national 
strategies or plans.*' While the EU and its member states continue to lag 
behind the US and China in AI, some reports suggest “the EU is catching 
up fast” and “AI is the most dynamically developing technology in terms 
of patent filing and start-up activity in Europe (outpacing the US)."* The 
focus on AI strategy development marked an increase in pledged national 
and EU-wide spending on digital and AI technologies. The EU committed 
to invest €20bn annually in AI for Europe's Digital Decade (2020-2030), of 
which €1bn annually are allocated through the Digital Europe and Horizon 
Europe programmes. In addition, €134bn are allocated to digital transforma- 
tion under the Recovery and Resilience Facility. This European increase in Al 
funding is driven by domestic need and reduction of strategic dependencies as 
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well as by the perception of a trend among leading great powers to subsidise 
technological (and particularly AI) adoption across their societies. The Com- 
mission and European states are also slowly beginning to create and diversify 
innovative funding alternatives to accelerate AI adoption, including venture 
capital funding. 

Since 2017-2018, there is visible EU progress in the field of AI strategy 
development and funding. To the degree a European approach to AI is 
emerging, it is primarily normative and regulatory, bordering on variable 
degrees of techno-nationalism, Eurocentrism, and European exceptional- 
ism.** It is driven by the ambition of asserting a qualitative European lead- 
ership in technology matters at the global level by exploiting first mover 
market integration, regulatory and hybrid standardisation effects. And it 
is marked by uneven progress between the civilian and the security and 
defence fields, closely reflecting the status of the broader EU integration 
process. 


AI and European defence 


There are great differences between European states with regard to AI adop- 
tion in defence. As recently emphasised by EU officials, the status quo “is not 
sufficient if only one or two Member States have [military AI applications]. 
They must be available across all EU Member States.”*° EU adoption of AI 
in defence has so far concentrated on three main lines of effort which are not 
always well coordinated. 


Political efforts 


The first line of effort is political. It resides within the Council, the European 
Parliament and, to a lesser extent, the External Action Service. Rather 
than facilitating a strategic common EU policy approach to AI in defence, 
different political processes remain siloed and fragmented. The first Euro- 
pean high-level exploratory consultation of digitalisation and AI in defence 
took place in August 2019, during the Finnish presidency of the Council and 
incentivised two parallel processes: an EDA process to raise awareness on Al 
in defence and an EU Military Staff process to accelerate defence moderni- 
sation through the digitalisation of defence. In 2020, the German presidency 
proposed an EU process on the “Responsible military use of artificial intelli- 
gence’ to agree a set of common norms and standards for the operational use 
of AI and the export of Al-enabled military systems. However, this process 
is not synchronised with any other ongoing Al work in the EU and has 
not yielded any results given the opposition of other member states." The 
European Parliament has explored the manifold role of AI in defence*® and 
external action.” However, this has done little to change its political position 
driven by a normative logic of AI adoption in defence rather than one fo- 
cused on shaping AI adoption towards achieving greater military advantage. 
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The Parliament continues to insist on strict regulation and even ban of AI- 
enabled autonomy in defence, including in cyber defence.>” 


Technical efforts 


The second line of EU effort is technical. It is concentrated within the European 
Commission, the EU Military Staff and the European Defence Agency and 
fostered within the EU defence initiatives,”! though there is no coherent 
strategy linking the capabilities developed in Permanent Structured Cooper- 
ation (PESCO) and European Defence Fund (EDF) projects to specific EU 
operational doctrine and clear metrics of greater operational capacity and 
interoperability necessary for greater strategic autonomy.” The publication 
of the 2021 EDF work program suggests greater European emphasis on Al 
and other EDTs,® which makes a common European approach an urgent 
necessity. 

The narrative around several loosely connected EU work strands promises 
to deliver such an approach. A 2021 EEAS Memo on the status of the Strategic 
Compass negotiations lists “strengthen[ing] the European Technological and 
Industrial Base including in particular an enhanced common EU approach to 
emerging and disruptive technologies in the security and defence domain” as 
one of three priorities in the “Capabilities basket.’°* However, the Strategic 
Compass document generically mentions emerging technologies in the con- 
text of long-term, next-generation capabilities investment. 

In 2021, the European Commission worked on a Roadmap on Critical Tech- 
nologies for Security and Defence which operationalised the Action Plan on Syn- 
ergies between Civil, Defence and Space Industries, notably by identifying means 
to incentivise and support civilian innovation with potential defence applica- 
tions. The Roadmap, published in 2022, guides further Commission action on 
technological roadmaps, in conjunction with the Commission’s EDTs Ob- 
servatory. However, even the Roadmap’s proponents admittedly do not have a 
long-term perspective on the utility of this tool.55 

Since 2019, the EDA has developed an AI in Defence Definition, Taxonomy 
and Glossary, an AI in Defence Narrative, and AI in Defence Action Plan which 
could be linked to capability development and collaborative European R&T 
projects. Based on these, the EDA has developed an AI Strategic Research 
Agenda and identified over 50 technology building blocks relevant to AI. 
And in 2021, the EDA started preparing an EDT’ Action Plan which will 
support EU states in “monitoring the EDT landscape in and outside of the 
EU” through horizon scanning and foresight and identifying and pursuing 
collaborative R&T projects “to avoid fragmentation and duplication.”°° 

The EDA’s 2020 Coordinated Annual Review on Defence (CARD) re- 
port indicates EU defence initiatives are too recent to effectively steer na- 
tional defence planning and it is worth asking whether empirical evidence 
supports the logic behind capability development in the EU, from the CDP 
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and OSRA to CARD, EDF and PESCO, in the context of military AI. 
However, the lack of convergence between different EU innovation initia- 
tives is not just a symptom of intensifying inter-agency power struggles to 
expand competencies and political clout. They are also evidence that a com- 
mon AI approach in defence is yet again failing to emerge among key EU 
institutions and member states. 


Operational efforts 


The third line of effort is at the operational level, within the EU Military Staff. 
It focuses on the overall objective of defence modernisation through the digi- 
talisation of armed forces. The latter is a multi-year defence modernisation effort 
towards network-centric warfare and network-enabled operations (a process 
Europeans started in the late 1990s) in which AI is a strategic enabler.°’ In 
this context, Al’s highest likely impact in Intelligence, Surveillance and Re- 
connaissance (ISR) and cyber defence applications.55 The added value of AI 
resides in enhanced situational awareness through multi-sensor information 
fusion and processing and decision-making assistance, which explains the 
European increased investment in sensor technologies. However, the AI use 
cases are not grounded in new operational concepts and doctrine but are 
treated as optimising accessories and layers to existing military capabilities. 

In addition, AI enables European armed forces to improve efficiency across 
three dimensions: to do things better, notably, to increase their military effec- 
tiveness, readiness, and manoeuvrability, reduce resources and operational 
footprint; to do better things, such as improving mission planning, execution, 
and decision-making; and to do new things, such as sharing a common operat- 
ing picture across the strategic, operational, and tactical levels, at national and 
multinational (EU) levels or sharing real-time information and coordinating 
across the civilian-military spectrum.” 

There is a disconnect between the EU’s political and military levels which 
obstructs a broader AI impact on strategic autonomy by putting a normative 
glass ceiling on innovation. The political level holds a deeply normative view 
of Al adoption in defence, with a desire to limit uses of autonomy and control 
technology diffusion. The technical and operational levels are deprived of 
any clear steer on adopting AI technologies to achieve a European military 
technological and operational advantage and of sustained political support for 
their plans. Meanwhile, there is no structured EU consideration of the im- 
pact of its normative and over-regulatory approach to AI on future European 
military advantage. Absent a convergent political and military approach to 
the integration of AI technologies in defence for the purpose of creating mil- 
itary advantage for the EU, AI will remain a distraction as far as EU strategic 
autonomy is concerned because algorithmic military power will remain out- 
side the reach of the Union. In the next section, I identify and explore four 
reasons which explain this situation. 
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AI for autonomy: can AI contribute to greater 
strategic autonomy in defence? 


Digital blueprints for technological autonomy 


W hether one subscribes to the narrow or the wider understanding of stra- 
tegic autonomy, technology is at the core of the concept. European Council 
conclusions,°” the EU Global Strategy,°! the Implementation Plan on Secu- 
rity and Defence, and the EDF regulation®? refer to strategic autonomy in 
the context of a sustainable, innovative, and competitive European defence 
technological and industrial base (EDTIB). The EDF specifically refers to 
the EDA’s Capability Development Plan and the Overarching Strategic Re- 
search Agenda, which define current and future capability and technological 
requirements, as guiding instruments in the selection and implementation of 
EU-funded defence projects. The same is true for the EDA’s AI in Defence 
Action Plan and Strategic Research Agenda, its work on Cyber Defence and 
its upcoming EDTs Action Plan. 

Technological autonomy in defence is closely linked to European defence in- 
dustrial competitiveness and it has been at the core of strategic autonomy 
since 2013. It refers to the EU’s ability to fund, develop, adopt, and inte- 
grate in defence capabilities critical technologies that are primarily (or exclu- 
sively) developed in Europe, are controlled by Europeans, and have secure 
supply chains. It is reflective of strategic European dependencies on raw ma- 
terials and rare earths for the development of their military capabilities® as 
well as on foreign technologies that are subject to external export controls.°° 

This marks an evolution in the way the EU conceptualises the relationship 
between AI, cyber power, and strategic autonomy in defence, which was only 
implicit prior to 2019. At the 2018 EDA Annual Conference, former HR/ 
VP Mogherini highlighted AI “is also a matter of security.”°” The May 2019 
Food for thought paper on Digitalisation and AI in Defence acknowledged the role 
of AI as enabler of greater digitalisation®® without specifically linking it to 
the objective of strategic autonomy. 

By 2021, the EU’s strategic thinking about the role of cyber and AI (as 
well as other digital EDTs) as indispensable elements of power, that shape the 
global balance of power, had evolved. The EU’s 2021 review of the Cyber 
Policy Framework and the Military Vision and Strategy on Cyberspace as a Domain 
of Operation prioritise the full integration of cyber resilience, cybersecurity, 
and cyber defence “into the wider area of security and defence”? alongside 
other emerging technologies such as AI. 

Furthermore, the EU Military Committee has developed a Strategic Imple- 
mentation Plan for the Digitalisation of EU Forces which sets a level of ambition, 
specific targets, and milestones for the digitalisation and interoperability of 
European armed forced and discusses the role of AI as an enabler of digitali- 
sation in defence. Finally, the Commission, the EDA and individual member 
states are trying to adapt their industry engagement strategies to enhance 
their access to cutting-edge technologies and accelerate adoption. In short, 
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AI is very much at the centre of European technological autonomy. The 
development of European AI technologies for defence can contribute to less 
European dependence on external actors for critical technologies associated 
with their military power. 

However, EU (and individual national) efforts are not the only determi- 
nants of strategic autonomy. Exogenous factors are increasingly important in 
relation to EDTs, as highlighted by the proliferation of the use of concepts 
like “the geopolitics of technology.” Among such exogenous factors 1s the re- 
lationship with private industry which is largely responsible for the rapid pro- 
gress and accelerated investment in both AI and cyber. Continued European 
dependency on digital technologies and infrastructure (e.g., military cloud) 
from non-EU countries and industry actors entails the Union’s freedom of 
action is inherently linked to its ability to build consensus with these external 
actors or risk tense geopolitical relations. 

The structure and operation of the global internet may also act to inhibit 
European strategic autonomy independent of or in combination with other 
EU digital strategic dependencies. A normative approach to cyberspace and 
AI integration in defence without the backing of a technological foundation, 
is feasible only so long as the technological decoupling between the US and 
China does not accelerate and the internet is split into American and Chi- 
nese digital sphere of influence, which some analysts believe it is already the 
case (i.e., splinternets). Geopolitical alignment patterns are also important 
influence networks. For example, geopolitical constructs like the Belt and 
Road Initiative (BRI) and emerging strategic cooperation between the US 
and Indo-Pacific allies increasingly act as ‘technological transmission belts’ 
and ‘techno-spheres of influence’ with geopolitical impact.”” 

Finally, while the EU’s regulatory power can shape market rules for AI tech- 
nologies and it has established processes to negotiate rules for the “Responsible 
use of military AI,” the Union cannot similarly shape global rules of opera- 
tional deployment for AI and cyber in defence by itself. As EU external action 
takes place predominantly in a multilateral context, this inherently means 
the EU will require a new perspective on tech-diplomacy and partnerships, ’! 
which goes beyond the economic and trade field and addresses key defence 
and industrial issues related to competitiveness. However, building effective 
and operational partnerships has not been a priority for the EU. And it remains 
unclear how much the Union’s approach will change in form and substance 
under the Strategic Compass. If the EU’s security and technology partnerships 
will be driven by the same normative logic of the cybersecurity and hybrid 
threats toolboxes rather than a geopolitical one, they will arguably not lead to 
any significant enhancement of the EU’s strategic autonomy in defence. 


Digital blueprints for decision-making autonomy 


Decision-making autonomy refers to the ability of the EU and European states 
to define their own strategic interests, collect and analyse data, and make 
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decisions independently of other international actors. The adoption of AI and 
the switch to “data-driven” policymaking requires further consideration of 
two aspects: what is the role of AI technologies in how EU decision-making 
in security and defence is taken and what are the AI technologies precisely 
used for in this process. Neither of these aspects is evident in the EU’s ap- 
proach to AI adoption in defence, which serves to underline the strategic 
difference between developing AI technologies and effectively employing 
them as instruments of statecraft. 

It remains unclear how and for what purposes the EU can use AI tech- 
nologies in decision-making to enhance its strategic autonomy. Based on 
the EU’s AI initiatives in defence, the priority is the automated analysis of 
growing volumes of data in (near) real time. The May 2019 EUMS paper 
on AI and the digitalisation of the armed forces references the use of Al to 
gather, fuse and process large volumes of data at ever greater speeds for the 
generic purpose of “information superiority.” According to its webpage, 
the European Space Agency (ESA) is already using algorithms to analyse 
satellite imagery, though not necessarily in defence-related scenarios. The 
EDA’s Action Plan on AI in Defence resulted in three priority use cases: Al 
for cyber defence, AI for ISR and enhanced situational awareness and AI 
for smart maintenance.’* The EU’s 2021 Cybersecurity Strategy refers to the 
use of AI to monitor and manage digital networks and develops the EU’s 
institutional structures that support cyber security, including through the 
creation of a Joint Cyber Unit, an EU cyber intelligence working group and 
a network of Al-enabled cyber centres across the member states. Beyond 
this, EU can better leverage AI for decision-making support and CSDP 
mission planning and execution. For example, the EEAS is exploring the 
notion of using Al-enabled dashboards for conflict prevention and early 
warning’? while the EUMS is interested in Al-enabled mission planning 
for different scenarios. 

At the political level, strategic autonomy has been linked with discussions 
about the reform of EU decision-making on security and defence. Specifi- 
cally, this refers to the proposal to adopt QMV in the Council and activate 
article 44 TEU on issues pertaining to foreign policy as a means of facilitating 
swifter, more agile and more unified EU positions and responses to interna- 
tional developments.’ Notwithstanding academic debates about the virtues 
of efficiency and legitimacy in EU security and defence decision-making, 
QMV and art 44 TEU are geared towards political efficiency by bypassing the 
current consensus rule. This is in recognition of the fact that in responding to 
crises, speed is a critical consideration whereas building consensus 1s often a 
time-consuming process that frustrates rapid response and leads to the mini- 
mal common denominator among the EU members on the type and scope of 
response. However, it’s unclear how AI can be leveraged alongside QMV and 
art 44 to deliver a more efficient EU “data-driven” decision-making pro- 
cess in foreign, security and defence policy. Indeed, integrating AI and Al- 
generated information into European political and military decision-making 
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may pose several challenges to the QMV model which need to be accounted 
for. Three categories of challenges are telling. 

First are procedural challenges, such as the origin and reliability of the data 
and algorithms used in the decision-making process and national and in- 
stitutional resistance to data sharing. The adoption of AI also adds a layer 
of complexity to military and civil-military decision-making by requiring 
a reconsideration of appropriate levels of decision-making for specific AI- 
supported actions and dedicated efforts to foster a shared civil-military oper- 
ating picture and threat understanding. 

Second, normative challenges should not be underestimated. These include 
whether the actionable information generated by AI systems is compliant 
with European values and it is trusted by decision-makers from different 
nations, with different access to comparable levels of intelligence and tech- 
nology. While it is true for NATO that “the sense of equality and co-decision 
among members could be at risk because of worries about accountability,” > 
this also applies to EU decision-making. Whether in cyberspace or in the 
physical domains of warfare, the adoption of AI implicitly creates require- 
ments for greater use of autonomy which conflict with different normative 
policies of many EU member countries. 

Third, there are important foundational challenges to consider. Some of 
these emanate from the mismatch between technological potential and the 
specific problem(s) AI technologies are meant to address. AI algorithms are 
modelled on optimising performance in specific contexts and instrumen- 
tally dealing with cognitive tasks. They are not built to support consen- 
sus and a common understanding which are the main challenges for EU 
decision-making. Arguably, Al-enabled decision-making assistance could 
challenge pluralistic decision-making among various European actors, with 
different interests and ideological preferences. For example, AI can reveal 
patterns of adversarial Russian and Chinese behaviour. However, it will not 
eliminate diverging national preferences or lead to a coherent EU policy on 
either of these rival great powers. 

Other foundational challenges emanate from the nature of the action and 
threat that is subject to decision-making. The EU’s decision-making model — 
regardless of whether it is QMV, art 44 or another — remains essentially reac- 
tive. The use of Al-enabled capabilities and enhanced situational awareness 
could make a predictive and/or pre-emptive model possible and it could even 
automate some defence responses. However, the EU is unprepared to fully 
leverage this technological potential without a radical shift in its strategic 
culture. 

Managing expectations about what type of European action AI would 
enable is important in understanding whether and how (much) AI technol- 
ogies support European strategic autonomy. Specifically, an important lim- 
itation for Al technologies in the context of EU decision-making is that it 
cannot generate or replace political will to act. In short, AI technologies will 
not cure Europe’s ‘bias for inaction.’ The risk of reductionist techno-centric 
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approaches cannot be discounted and absent an EU common approach to 
AI in defence, these challenges will not be easy to bypass regardless of the 
decision-making format. 


Digital blueprints for operational autonomy 


Towards an EU digital military level of ambition? 


Since 2016, the Union sought to address some of its main challenges through 
PESCO, the EDF, the CARD, EU-NATO cooperation and the Strategic 
Compass. A well-equipped and operational rapid reaction force, the consoli- 
dation and competitiveness of its defence industrial base, the development of 
modern military capabilities and the development of an autonomous military 
command and control and planning tool remain critical to (self-) perceptions 
of European strategic autonomy. In other words, operational autonomy and 
military capabilities are key to European strategic autonomy. 

Operational autonomy refers to the ability of European states to jointly de- 
velop competitive modern military capabilities, in sufficient numbers, levels 
of interoperability and readiness and to be able to successfully launch, conduct, 
and sustain military operations across the full spectrum from low-to-high in- 
tensity warfare independent of external (i.e., American / NATO) support. 

The creation of an EU operational and well-equipped rapid reaction force 
(rather than ad hoc multilateral coalitions) is regarded as a “clearly decisive step 
towards European defence.”’° However, the EU’s level of military ambition 
does not match its policy rhetoric. In the context of the Strategic Compass, 
several EU member states agreed to launch a joint “first entry” force of 5000 
troops, possibly based on the activation of the EU Battlegroups.’’ In addition, 
to enhance its operational autonomy, the EU 1s also developing an EU Full 
Spectrum Force Package, an EU Strategic Reserve Concept, an EU CSDP 
strategic stockpiling for military CSDP Operations and Missions as well as 
an EU Concept for Military Command and Control. Nevertheless, to put 
things in strategic perspective, the EU’s 5000-troops rapid response force (es- 
sentially, the EU’s version of VJTF), which is expected to undertake two or 
more simultaneous operations of various scope and intensity, is smaller than 
the American contingent securing Kabul airport during the August 2021 
withdrawal and is roughly similar in size to the estimated size of Russian 
troops operationally deployed in Syria. Furthermore, it is unclear how and 
whether the EU will overcome significant force generation, common fund- 
ing, and decision-making challenges in implementing the first entry force.’® 

The scope of the EU’s military level of ambition — and therefore, the scope 
of its operational autonomy — are linked to the ongoing process in the Stra- 
tegic Compass to 


reflect on realistic contingencies in light of the ring of instability and 
tension around Europe and beyond. In essence, in addition to preparing 
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for one major contingency (which has subsequently been complemented by 
several other illustrative scenarios), the European Union now needs to 
focus much more on preparing to undertake simultaneously several smaller and 
medium-sized operations — and not only on land, but also at sea and in the air? 


This is not new — it essentially describes the EU’s military level of ambition 
under the Headline Goals which have yet to be achieved. The caveat, of 
course, is that under the Headline Goals, EU operations were not necessarily 
based on scenarios that required either long-distance power projection or the 
EU’s ability to operate in highly contested and denied environments like the 
Indo-Pacific, Eastern Europe, or the High North. 

In addition to national efforts on integration of AI in defence, the EU 
Military Committee has developed a Strategic Implementation Plan for the Dig- 
italisation of EU Forces which sets a level of ambition for the digitalisation of 
European armed forces, including the development of a full spectrum, scal- 
able digitalised force package,®” and provides an implementation roadmap. 
According to the EU Military Staft's definition, digitalisation of defence refers 
to the application of multiple information technologies (including AI) to ac- 
quire, process, disseminate, and use information across the multi-domain bat- 
tlespace, by networking sensors, capabilities and forces and decision-makers 
and enabling the achievement of information superiority.*! Therefore, the 
concept of digitalisation of defence highlights the interdependence between AI 
and cyber power. 


Digital futures: AI and cyberwarfare 


It is worthwhile briefly exploring how technological convergence between 
cyber and AI technologies is relevant for modern warfare and European stra- 
tegic autonomy. 

AI technologies used in military applications have the potential to rev- 
olutionise the way wars are fought. In cyberspace, the frequency of ad- 
versarial interaction, the variety of actors and the challenge of attribution 
translates into the added value of increasing automation of cyber defen- 
sive and offensive tools. Cyber is not just a new domain of operations, but 
cyber power is “a new form of power projection” for great powers that 
exploit both the military and civilian capabilities, notably “sometimes em- 
ploying companies under their jurisdiction and control as their agents.” 8? 
These perceptions are grounded in the higher frequency and intensity of 
cyber-attacks during the COVID-19 pandemic, including but not limited 
to the Solaris Winds Orion attacks. They are also grounded in the strategic 
great power competition which is altering the structure of the internet and 
putting a premium on securing data as a strategic asset. Countries such as 
Russia, China, Iran, and North Korea control the digital sovereign space, 
notably cyber capabilities, the national internet, digital infrastructure, and 
data flowing through them. 


92 Simona R. Soare 


The relationship between AI technologies and cyberspace is one of inter- 
dependence. As a digital category of technologies, cyberspace is the primary 
domain for AI. As such, this paper considers both the role of AI technologies 
in cyberspace for cyber resilience, cyber defence, and cyber offense capabili- 
ties as well as the role of cyber security and cyber defence for the viability of 
deployed AI military applications. 

First, AI technologies contribute to the automation of cyber offensive 
tools.5% The relevance of Al-enabled cyber offense stems from “new moti- 
vations for operations within the cyber domain” which “differ dramatically 
from the more conventional digital threats,’”** creating “a new class of persis- 
tent threat tools” and actors? and challenging current assumptions on cyber 
offense-defence balance and cyber conflict prevention. Such transformations 
are grounded in the greater sophistication, speed, reach (i.e., attack surface), 
adaptability, and complexity of Al-enabled cyber capabilities for both de- 
fensive and offensive purposes, all of which are beyond the human speed of 
reaction and capacity to control. Al-enabled cyber offensive tools could seek 
to illicitly access data and learn from it the vulnerabilities of target systems, or 
it can adapt throughout the mission to select from a range of options on how 
to proceed and infect the target network. However, “the prospect of subvert- 
ing Al-driven security functions (...) incentivizes operations in cyberspace 
beyond in-domain effects and outcomes.”®° In other words, a cyber-attack 
could result in physical effects. Examples include cyber-attack on armed un- 
manned vehicles used to attack civilians or friendly troops. 

Second, for cases in the virtual and physical domains, AI technologies hold 
the same promise of improving cyber resilience and cyber defence through 
the deployment of automated network monitoring and early detection and 
response agents. Though progress has been slower, AI algorithms can be 
trained to detect and automatically patch software vulnerabilities in human 
and digitally enabled code writing,“ or they can detect, neutralise, and re- 
spond to a cyber-attack on physical military platforms as in Purdue Univer- 
sity’s AI2I project. 

Third, AI systems are particularly vulnerable to cyber-attacks, either 
through “input attacks” which mislead the AI by skewing its pattern recogni- 
tion through deceptive measures or through “poisoning attacks” which target 
and corrupt the code of AI algorithms and the data they use.5% Examples of 
Al-enabled cyber defence systems are proliferating — Automatic Intelligent 
Cyber Sensor and Enterprise Immune System are just two such examples. 

However, a note of caution is warranted. AI systems used in cyber are still 
faced with technological maturity challenges as well as with networking, 
infrastructure, and resource challenges?’ and their efficiency varies depend- 
ing on the exact model used (deep or machine learning). Manned or un- 
manned complex military platforms such as submarines, fighters, drones, and 
armoured vehicles may require teams of autonomous cyber defence agents to 
be operationally deployed on the battlefield. Machine-to-machine interac- 
tion protocols and autonomous cyber agents teaming protocols, bandwidth, 
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electrical and computing power consumption are important challenges to the 
rapid adoption of operationally deployed Al-enabled cyber agents. Particu- 
larly challenging is that autonomous cyber agents may need to operate for 
longer periods of time with limited or no human in the loop when deployed 
in highly contested battlefields. As a result, their activity could cause the 
malfunction of military systems they are meant to protect,” and such mal- 
functions could take longer to discover absent close interaction with human 
operators. 

In other words, while technology maturity levels for state-of-the-art Al 
may not enable their deployment and efficient operation in real-life oper- 
ational conditions in the medium term, fully automated cyber capabilities 
are not just possible in the foreseeable future but may indeed become alto- 
gether necessary. As likely “primary cyber fighters on the future battlefield” 
stealthy, resilient, and multipurpose autonomous cyber agents will be critical 
for cyber defence as well as for the cyber security of other actors, including 
forces and military platforms on the battlefield.” 


Outstanding questions 


In the context of new operational requirements of algorithmic warfare, EU 
planning efforts in AI and cyber are a step in the right direction. However, 
there are several outstanding questions regarding the European adoption of Al 
in defence, including whether under the given circumstances the EU repre- 
sents the appropriate format and level for implementing and conducting algo- 
rithmic warfare. Ongoing efforts fall short of a common European approach 
to emerging technologies or a military strategy which would fully integrate 
cyber and Al-enabled capabilities into CSDP multi-domain operations, iden- 
tify capability targets and milestones, direct investment and defence plan- 
ning, utilise autonomous European intelligence about AI and cyber threats 
posed by strategic rivals and clarify military and political decision-making 
procedures around the use of AI and autonomy in cyber, space and the phys- 
ical domains of operations. 

On the policy and planning front, it is unclear whether EU efforts to in- 
centivise and support the development of Al-enabled capabilities will be suc- 
cessful across a wide range of European armed forces. Equally, it is unclear 
whether the current EU military level of ambition, doctrine and training are 
adequate for AI integration, for the requirements of cyber security and resil- 
ience of Al-enabled military applications, and issues related to industry and 
the security of digital supply chains. There is currently no adequate under- 
standing of the required level of cyber security in European mission critical 
systems and platforms. On the technical side, it is unclear whether the EU 
member states can overcome their capabilities fragmentation which poten- 
tially puts European strategic autonomy at risk.?? 

Even relatively low-hanging fruit such as Al-enabled logistics and main- 
tenance systems for multinational operations are challenging for European 
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states because they operate in a low-trust environment. Sharing data into 
common “data lakes’ to facilitate the development of common capabilities 
and platforms used in CSDP missions and operations is still not possible 
and whatever national Al-enabled maintenance capabilities are currently 
under development would not be deployable for security reasons,’ albeit 
enhanced space-based communications capabilities could change this in the 
future.%5 Significant concerns also exist about the lack of European deploya- 
ble advanced C4ISR battle networks’? which means the utility of individual 
Al-enabled capabilities is rapidly undermined by exceeding security risks. 


Autonomy by partnership? AI and EU-NATO cooperation 


The Implementation Plan on Security and Defence argues “Europe’s strate- 
gic autonomy entails the ability to act and cooperate with international and regional 
partners wherever possible, while being able to operate autonomously when and 
where necessary.” Nevertheless, this poses a dilemma and a trade-off for 
the EU in developing and operating Al-enabled capabilities and cyber power 
autonomously or in cooperation with partners. 

The EU-NATO cooperation and EU-US cooperation are cases in point. 
European strategic autonomy requires greater operational capability, includ- 
ing autonomous command and control. Nevertheless, in the context of the 
digitalisation of defence, some EU officials are keen to proceed with the 
full alignment of standard operating procedures with NATO’s Framework 
Network and the Federated Mission Concepts which are currently blocked 
by Turkey.” The EU requirements for military mobility were established 
based on NATO minimal requirements for infrastructure. This points to 
great interdependence between the EU and NATO and cyber and EDTs are 
no exception. 

EU and NATO representatives signed the 2016 and 2018 Joint EU-NATO 
Declarations which cover cyber, hybrid, and capability development as prior- 
ity areas of cooperation. Cyber is one of the areas of EU-NATO cooperation 
with the most consistent reported progress over the past five years,” across 
both qualitative and quantitative assessment metrics. NATO’s own efforts in 
the field of developing cyber doctrine, structures, capabilities, and a common 
strategic culture span almost two decades. Lessons learned from NATO’s 
efforts to integrate cyber into its operations, protect its own critical networks 
and infrastructure and make decisions on deploying cyber effects are fully 
relevant to the EU’s efforts. A case in point is the differentiated integration of 
defensive and offensive cyber effects in NATO operations. 

As AI technologies become increasingly incorporated into cyber capabil- 
ities, tasks and operations, closer EU-NATO cooperation on AI uses in the 
cyber domain seems a natural next step. The foundations for such coopera- 
tion already exist. Both NATO and the EDA (and ENISA) have explored the 
role of AI and automation in cyberspace. In 2016, the NATO Science and 
Technology Organisation (STO) established a research group on Intelligent 
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Autonomous Agents for Cyber Defence and Resilience which proposed a refer- 
ence NATO architecture for Autonomous Intelligent Cyber-Defence Agents 
based on their anticipated key functions and technical requirements as well 
as a roadmap towards their adoption. 10° EDA projects such as Cyber Defence 
Technology Landscaping project, CYSAP, CHESS, and GARD seek to sup- 
port EU member states in using Al technologies for cyber threat mapping, 
enhanced automated network resilience and human-machine cooperation 
for early detection of cyber incidents. Similarly, AI for cyber defence is one 
of the first pilot projects under both NATO’s 2021 AI Strategy and one of 
three priority actions under the EDA’s Action Plan for AI in Defence. This 
is a substantial foundation for closer EU-NATO cooperation in Al-enabled 
cyber defence. 

However, as the EU and NATO negotiate the third EU-NATO Joint Dec- 
laration which will reportedly focus on EDTs and climate change coopera- 
tion, the level of ambition for EU-NATO cooperation remains vague.” TA 
2021 German-Dutch Food for through paper advocated a new [EU-NATO] 
“joint declaration” establishing “stronger political consultations” of a joint 
informal working group on EDTs.102 

Finally, closer cooperation also entails a deliberate effort on the part of 
the EU and NATO to avoid competition." Both EU and NATO defence 
innovation landscapes are becoming more complex, with new bodies and 
instruments recently established. In May 2021, the European Council called 
for “reinforcing the role played by the EDA in fostering defence innova- 
tion including disruptive technologies”!°* and in June 2021 NATO leaders 
agreed to establish a Defence Innovation Accelerator for the North Atlantic 
(DIANA), backed by a dedicated NATO Innovation Fund (NIF) report- 
edly worth $1bn. Both the EU and NATO’s new innovation structures are 
experiencing early challenges — the US and France, two of the lead innova- 
tors in NATO are reluctant to join DIANA and NIF, whereas the push for 
the EDA’s greater role in defence innovation is almost exclusively backed by 
Paris. Fears of duplication (especially on the European side) are pervasive — 
though there is little evidence of ongoing efforts to ensure complementary by 
design between these complex innovation structures and even less evidence 
of why duplication in research and innovation is necessarily problematic. The 
current focus in the EU and NATO frameworks seems to be on form rather 
than substance with the (still unproven) expectation that once appropriate 
innovation structures are in place, substance — and strategic results — will 
follow. Along with the challenges in transatlantic defence cooperation on Al 
and other emerging technologies!’ this is a reminder that (the degree of) 
European strategic autonomy may not necessarily be a choice, but a necessity. 


Conclusion 


Europeans are arguably still in the incipient phases of their AI efforts and the 
bulk of the difficult tasks of developing and widely adopting AI technologies 
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still lie ahead. The EU is off to a good start in planning its technology policies 
and seeking cross-fertilisation as a long-term strategy of technology competi- 
tion with and strategic autonomy from other great powers. 

While AI and cyber convergence has significant strategic implications for 
European security, this paper proved that a techno-centric European strate- 
gic autonomy is a distraction. Al adoption in defence will add a new layer of 
complexity to a complicated multinational and institutional landscape, with- 
out a clear perspective on how it will be translated into an effective tool of 
statecraft for greater strategic autonomy. This finding is substantiated by em- 
pirical evidence across four dimensions of strategic autonomy: technological, 
operational, decision-making autonomy and defence partnerships. 

This paper has barely scratched the surface of the research needed on the 
impact of AI on European strategic autonomy in defence. It was not meant 
as an exhaustive ontological analysis of Al and European strategic autonomy 
or as a dissuasive narrative towards greater EU efforts to integrate AI, cyber 
effects into defence. Rather it was intended as a pragmatic analysis of four 
critical dimensions of strategic autonomy and Al's impact on them in rela- 
tion to cyberspace and other domains of warfare. More research is needed to 
understand how or whether EU planning translates into qualitatively differ- 
ent Al-enabled capabilities, how differentiated AI adoption among the EU 
states influences their collective capacity for common action and how the EU 
employs its ‘algorithmic power’ across strategic partnerships, networks, and 
channels. 

The research findings of this paper have theoretical and conceptual impli- 
cations beyond the European context. Its findings on the conceptualisation of 
the empirical relationship between technology and political autonomy for ac- 
tion is potentially relevant across a wide range of international organisations, 
alliances, and multilateral collaborative fora of different levels of institution- 
alisation. It has direct bearing on understanding the role such multilateral 
actors enabled by technology can play in great power competition. It also 
serves as a reminder of the perils of reductionist techno-centric approaches 
to strategic autonomy which underestimate the complex challenges of trans- 
lating technological and innovation potential into actual tools of power in 
international politics. 
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1 Such technologies include artificial intelligence (AI), quantum computing (QC), 
big data analytics, more resilient cyber networks, cloud and edge computing, 
sensor technologies, next generation (tele)communications, space technologies, 
chip and semiconductor technologies, and their supply chains and functional 
integration. 
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5 The middleware dilemma of 
middle powers 


Al-enabled services as sites of 
cyber conflict in Brazil, India, and 
Singapore 


Arun Mohan Sukumar 


Introduction 


Although no accepted definition of the term exists, “middle powers” includes 
those countries that exercise a high degree of economic sovereignty, ! and 
strongly influence regional developments through their political, economic, 
cultural, or military capabilities. Middle powers may also influence rule- 
making in specific domains of global governance. Notably, these countries 
tend to favor the status quo of the liberal international order, preferring to 
engage multilateral regimes and seek accommodations from them as needed, 
rather than challenging those regimes or the broader, hegemonic interests 
that underpin them.” “Middle powers” encompasses both developed and de- 
veloping countries. Indeed, the term is as much a reflection of states’ material 
capabilities as it is of “normative and behavioral criteria." Middle pow- 
ers aspire to maintain and elevate their international status. Technological 
leapfrogging — the adoption by developing countries of frontier technologies 
for governance, skipping in the process older-generation technologies that 
are either resource-intensive or unscalable — figures prominently in this pur- 
suit of status.“ Leapfrogging, whether through the adoption of GSM teleph- 
ony, IPv6, or 5G, has not just been viewed as a sustainable path to economic 
prosperity. It is also a totem of empowerment for middle powers, allow- 
ing them to participate on equal footing with Great Powers on Research & 
Development, standard-setting, and application of new technologies for their 
own requirements. 

Artificial Intelligence and Machine Learning (AI/ML) represent one such 
frontier technology. Since the publication of the world’s first national AI 
strategy in 2017 by Canada — a self-described “middle power”? — no less 
than 54 countries have either announced or are in various stages of declar- 
ing their own national strategies.° Several of these strategies, especially those 
drawn by middle powers, emphasize the importance of AI in leapfrogging 
hurdles to the delivery of services in sectors such as healthcare, education, 
transportation, and e-commerce. In particular, they underline the need to 
take advantage of vast troves of data generated by their population to train 
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predictive algorithms and develop ML models for the efficient delivery of 
said services. Given the costs and limited availability of high-skilled work- 
ers to train healthcare professionals, lawyers, and educators, among others, 
emerging markets may seek to replace them with Al-enabled ‘bots.’ 

A review of national AI strategies suggests states may pursue one of two 
paths to promote innovation and adoption of Al-enabled services. Some 
countries may choose to “make public datasets available” for market players 
to develop proprietary Al-enabled services, and concurrently, set up regu- 
latory sandboxes to pilot those products.’ Other states may opt to leverage 
their Digital Public Infrastructure (DPI), i.e., public technical standards and 
protocols that allow third parties to access personal data of citizens or an- 
onymized data, with clearly defined guidelines on the nature of data that 
can be shared, the duration of data sharing, as well as permissible use-cases.® 
Such infrastructure, which can be considered “middleware,” then becomes 
the conduit for access of data to train AI/ML models. The “middleware” 
model is likely to be preferred by many states as it allows for more granular, 
revocable, and regulated data-sharing, rather than one-time access to public 
databases. Middle powers in particular may condition the use by market play- 
ers of such middleware on their affordable provision of Al-enabled services in 
sectors such as health and education. 

Examples of such middleware infrastructure developed by middle pow- 
ers include protocols that standardize data-sharing (for example, Personal 
Health Records in Japan)? common and interoperable railroads for digital 
transactions (PayNow in Singapore,” Unified Payments Interface in India, 
etc.), or unique digital identifiers, often validated by biometric markers 
(Aadhaar in India,” MOSIP in Philippines," e-ID system in Estonia,“ etc.). 
The leveraging of middleware DPIs towards Al-enabled services is in the 
early stages of conceptualization and implementation at the time of writing. 
Nevertheless, it is highly likely that middle powers that have made con- 
siderable advancements in building such infrastructure — Australia, India, 
Brazil, Norway, Japan, South Africa, and Singapore, to name a few — will 
rely on them to promote innovation and at-scale adoption of Al-driven 
services. Indeed, the high degree of standardization in data collection, la- 
belling, and sharing through middleware DPls makes it easier to develop 
and train ML models. DPIs also allow regulators to calibrate the nature of 
data collected, allowing in some cases for the sharing of scarce personal and 
population-level information, and minimizing the collection of potentially 
harmful information in others. These reasons make it probable that public, 
middleware infrastructure will be used to develop Al-based applications 
and services across the world. For instance, it is worth highlighting how 
the COVID-19 pandemic has accelerated the adoption of Al-enabled diag- 
nostics that rely on data from contact-tracing applications or chest imaging 
databases built by states. This trend will likely continue in the future, 
especially in middle powers that have created centralized national databases 
faster than advanced economies. 
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The development of Al-enabled products and services that rely on such mid- 
dleware infrastructure also creates a dilemma for states. DPIs are heavily re- 
liant on Application Programming Interfaces (APIs) that lay down technical 
specifications for data collection and sharing. In some cases, the middleware 
in question is nothing but an API, i.e., a few lines of code that allow public 
and private players to connect with each other and share data. Payment rail- 
roads are a common example of such APIs. In other instances, APIs are neces- 
sary to allow external applications to connect to the public infrastructure and 
retrieve data from its servers. With respect to Al-enabled services or products, 
APIs will be key to ensuring that ML models developed by third parties have 
access to the right data parameters in order to train their algorithms. Essen- 
tial as they are to the adoption and scaling of digital infrastructure, APIs are 
also vulnerable to sophisticated cyber attacks. By their very nature, APIs are 
designed to facilitate the seamless integration of third-party applications with 
databases, cloud services, virtual networking functions, etc. The emphasis on 
ease of access has often come at the cost of secure API design.!° In the case of 
Al-enabled services that run on public datasets or middleware infrastructure, 
this vulnerability is compounded by the fact that there is minimal human 
supervision of the interaction between the service and the user. As a result, 
attacks on Al-enabled services through APIs could seriously impair both the 
availability of the service as well as its predictive accuracy and effectiveness. 
Attacks against the availability or integrity of Al-enabled services may thus 
have the effect of undermining public confidence and trust in them, espe- 
cially in developing countries and middle powers. 

To be sure, APIs are potent vectors for cyber attacks wherever they are 
deployed. With API ‘calls’ comprising over 80% of the Internet’s platform-led 
traffic, such vectors are omnipresent.!” The role of APIs in linking DPI to 
Al-enabled services, however, makes them an even more lucrative target for 
state and non-state actors who seek not only entry into critical national data- 
bases, but also disrupt autonomous and minimally supervised platforms that 
provide essential services. 

This chapter highlights how Al-enabled services running on public, digital 
infrastructure could emerge as vectors of cyber conflict. States face a difficult 
choice between opening up national databases or other public infrastructure 
to third parties in order to promote AI innovation atop its data and risking 
not only the security of those databases but also that of critical, even lifesav- 
ing, services. The “middleware dilemma” is most acute for middle powers, 
especially large emerging markets that are undergoing rapid industrialization 
but find themselves constrained by resources or capital to provide essential 
services at scale. As a result, they are compelled to turn towards digital or 
digital-enabled services, including AI services. 

The chapter is organized as follows: section two explores the ‘middle- 
ware dilemma’ in detail, highlighting security risks associated with API 
deployment, and their increasing role in the training of AI/ML modes. Sec- 
tion three outlines efforts by three middle powers — Brazil, Singapore, and 
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India — to make available public data and digital infrastructure through APIs 
to promote AI innovation in their healthcare sector. Drawing on these na- 
tional models and strategies, Section four presents an overview of threats and 
vulnerabilities faced by states in securing such services and their underlying 
infrastructure. 


APIs and the “middleware dilemma” 


The internet is witnessing unprecedented ‘API-fication.’ APIs are lines of 
code that allow software to communicate with each other, and thus facili- 
tate greater connectivity and interoperability among the network, data, and 
application layers that make up cyberspace. With digital environments be- 
coming increasingly heterogenous — most businesses and enterprises today 
delegate routing, data processing, and even cybersecurity functions to vir- 
tual networks and cloud services located halfway across the world — APIs 
have become crucial to the smooth functioning of critical internet services. 
Through their enabling role in the retrieval and processing of data at the 
application/device level, APIs have also been instrumental in realizing the 
“platform economy,” as it is known today.'® For the same reason, APIs have 
been key to the rapid expansion and proliferation of federated and central- 
ized databases across the world. The DPI developed by middle powers such 
as Australia, Brazil, and India, to name a few countries, too depend on APIs 
for their implementation and use. Indeed, regulatory tools such as the 2018 
Revised Payments Directive (PSD2) in the European Union and the 2020 
Consumer Data Right Act in Australia require even private actors to provide 
standardized APIs so that user data is interoperable and seamlessly accessible 
by all authorized parties. 

Despite their popularity, however, API security still leaves much to be 
desired. APIs have earned notoriety in recent years as attack surfaces for 
data breaches, identity theft and account takeovers, ransomware injections, 
IoT exploitation and DDoS attacks, among others.!? One estimate suggests 
API attacks will emerge as the leading vector of cyber attacks by 2022. The 
security considerations involving APIs are three-fold: first, the widespread 
use of APIs results in a crowding of digital networks and infrastructure by 
third parties, making it difficult to manage or monitor the proliferating end- 
points. Indeed, network administrators have no ‘over-the-horizon’ visibility 
with respect to third-party applications or devices that are constantly pinging 
their databases or infrastructure with API calls. Second, APIs may be devel- 
oped by actors across the network, but there are no clear frameworks for ac- 
countability and remedial action in case of API-enabled attacks.” The poor 
maintenance and updating of APIs has even contributed to the phenomenon 
of ‘zombie’ APIs that continue to be functional (and potentially leak data) 
although their developers have long abandoned their active use.7! Finally, 
a culture of data maximalism — “when in doubt, collect” — pervades API 
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design, with the result that API attacks often result in “excessive data expo- 
sure”? of users. A design culture favoring ease of access has also resulted in 
the neglect of security evaluations in the development cycle, although there 
is more awareness among API programmers today than even the recent past. 
On account of these factors, threats to API security have risen in severity 
and sophistication. The Open Web Application Security Project Foundation’s 
(OWASP) annual ‘Top 10’ rankings of API security threats — considered a 
benchmark among market players and cybersecurity researchers alike — has 
consistently identified the following as high-priority concerns:”? 


a Code injection, i.e., pinging the API with malicious code that allow 
unauthorized actors to retrieve, manipulate, or destroy user data; 

b Broken authentication, i.e., the use of APIs for credential stuffing or 
brute force attacks that permit malicious actors from taking control of 
user accounts associated with a service or application; 

c Man-in-the-Middle attacks that take advantage of poor encryption pro- 
tocols (at rest or in transit) to retrieve highly sensitive user details; 

d Insecure design of APIs that lean on legacy methods to recover user cre- 
dentials, retrieve data, generate error messages, etc., without adequate 
threat modeling. 


Given the nature and gravity of such threats, the use in particular of APIs 
that allow third parties to connect and retrieve information from middleware 
infrastructure presents a major cybersecurity concern for states. In the case of 
many middle powers, especially large developing countries, the government 
plays an important role in shaping the digital economy. The state in question 
may want to share data with market players in a bid to boost private innova- 
tion. While APIs present a relatively easy and seamless way for many states 
to create middleware that collects data or retrieves it from existing databases, 
they must balance such convenience against the risk of losing highly sensi- 
tive information. In some instances, only states have the legal imprimatur to 
collect certain types or categories of data from citizens, and the possibility 
of leakage or unauthorized exposure of such data (for example, biometric or 
health data) to third parties through APIs is high. 

The concern that APIs may be vectors for cyber attacks and indeed, cyber 
conflict, is compounded in the case of Al-enabled services. States may allow 
the use of APIs specifically to promote AI/ML innovation on public data- 
bases or infrastructure in a number of ways. Governments could lay down 
specifications and protocols for retrieving data either from databases or di- 
rectly from citizens. Once data is collected through such traditional, “dumb” 
APIs, it is left to the third party to anonymize the data and use it to train their 
proprietary ML models. Alternatively, states could provide “clean rooms” or 
closed environments where personal and population-level data is anonymized 
and training models built without actual transfer of data. Such a model relies 


114 Arun Mohan Sukumar 


on advancements in secure multi-party computation that allows for train- 
ing of algorithms without having to share private data.?* And finally, states 
could open up their infrastructure to third party Machine Learning APIs 
(ML APIs) that are used by start-ups, enterprises, and public agencies alike. 
Indeed, this third option may emerge as a popular one for many market play- 
ers who do not themselves have the capacity or resources to train ML models, 
but have innovative Al-enabled services to offer. The widespread adoption of 
cloud computing has boosted the popularity of ML-as-a-Service: Al-enabled 
products and services have increasingly begun to offload data processing and 
training of algorithms to cloud-based ML services such as Google, AWS, and 
Azure. Their APIs perform a number of critical functions, providing both 
‘off-the-shelf? and customizable neural networks to third-party applications. 
For start-ups that want to train their own algorithms on cloud-based services, 
ML APIs are invaluable for data labeling, maintaining registries of training 
models, and for periodic audit of those models.25 

These methods of using APls to facilitate third party access and innova- 
tion in Al-enabled services are not without risks. The security considerations 
and threats involving APIs in general have already been documented in this 
section, and need not be repeated here. Such threats are, however, more pro- 
nounced with respect to the use of APIs to train ML models. With greater 
volumes of data being called by APIs for training purposes, they become 
lucrative targets for state and non-state adversaries. A major concern with the 
use of traditional and ML APIs is their handling of data, and the measures 
taken by states as well as private actors to not only anonymize training data 
but also minimize risks of subsequent de-anonymization.~° In many develop- 
ing countries, judicial and regulatory capacity to address de-anonymization 
risks breaches may be limited, as a result of which its resolution could be 
entirely dependent on voluntary, technical steps taken by market players. 
Without effective safeguards to prevent de-anonymization, APIs could be 
exploited by adversaries to capture highly sensitive details from training data 
about the population. 

ML APIs have surged in popularity, especially in the aftermath of the 
COVID-19 pandemic. With businesses and NGOs moving their operations 
online, the demand for Al-enabled audio/video, text-based, and Natural 
Language Processing (NLP) services has increased significantly. The health 
sector has seen perhaps the biggest transformation during this period, as wit- 
nessed by the move towards predictive diagnostics and ‘health bots’ that per- 
form remote consultation. The rapid rise and adoption of ML APIs raise the 
concern that their software design may sidestep security considerations in 
favor of scale and ease of access. It is not simply the secure design of ML APIs 
that matter, but also their use by developers or services who are new to using 
‘off-the-shelf’ tools for training ML models. As Wan et al. note, ML API 
misuses have already become commonplace, because start-ups or businesses 
are not fully aware of attributes of ML tools offered by cloud services like 
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Amazon or Google.” Their study of 360 applications that relied on ML APIs 
found that developers routinely called the wrong API — for e.g., many apps 
confused “image classification” APIs with “object detection” APIs, the latter 
being used to identify objects within an image — which affected the accuracy 
and effectiveness of the service.”* Additionally, many developers also inter- 
preted the predictive results delivered by ML APIs incorrectly, mistaking 
probabilistic assessments for binary (‘yes or ‘no’) results.2? A poorly under- 
stood and utilized ML API ecosystem is ripe for exploitation and disruptive 
cyber attacks. 

As more applications and services rely on ML APIs to retrieve and train 
data through DPI, states will thus be confronted by the challenge of securing 
that data against API design flaws, improper use, and exploitation. Train- 
ing models not only require large datasets, but are also in need of constant 
updates both to the ML APIs as well as the data itself. As Chen et al. note, 
the predictive performance of ML APIs can grow “significantly worse over 
time” even when they rely on the same datasets.* Routine updates both to 
the API (by the cloud-based provider) and the training data are crucial to the 
model’s effective performance. Unfortunately, such a highly dynamic envi- 
ronment also increases the chances for MITM attacks that may be carefully 
disguised as API updates or requests for new data. Cloud service providers 
have been criticized in the past for considering security as an “externality,” 
and shifting the loss from cyber attacks on to the users.”! If they adopt the 
same approach with respect to ML APIs, especially those that ‘call’ public 
infrastructure, states may be constrained to address cyber attacks on their 
infrastructure and Al-enabled services quickly and effectively. A 2017 review 
of iOS and Android developer guidelines found many aspects of application- 
layer security on these platforms to be insufficient or only partly aligned to 
OWASP standards.*? With no human supervision of interactions between 
Al-enabled services and their users, similar vulnerabilities in ML APIs could 
be exploited to disastrous consequences. 

In summary, vulnerabilities associated with traditional and ML APIs could 
result in the misuse, manipulation, and even denial of Al-enabled essen- 
tial services that rely on them. Given deficiencies in secure API design, and 
in many instances, their poorly understood application with respect to core 
functions, API-driven middleware could be prime targets of strategic adver- 
saries in the event of conflict. Given these concerns, it is worth examining 
the different approaches of middle powers with respect to the adoption of 
APIs for Al-enabled services in critical sectors, and the possible security re- 
percussions of those API-led models. 


Case studies: Brazil, India, and Singapore 


The critical sector of healthcare has been identified by several states, includ- 
ing middle powers, as ripe for technology leapfrogging. Emerging markets 
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and developing countries with large populations have historically struggled 
to train medical professionals whether in the field of diagnostics or healthcare 
services. With a view to address the lack of skilled professionals, governments 
have turned to digital healthcare services. Furthermore, the COVID-19 pan- 
demic has catalyzed rapid advancements in digital health, including in the 
development of Al-enabled diagnostics and services. However, digitizing 
sensitive health data of populations and rendering them accessible to third 
parties — via API-based middleware — raises the possibility of such informa- 
tion being compromised or corrupted by adversaries. 

The following section outlines recent and ongoing efforts by three middle 
powers — Brazil, Singapore, and India — to make available public health data 
via APIs for external developers, including of Al-enabled applications in the 
sector. In particular, it emphasizes those historical and institutional reasons 
why these states have chosen to pursue three different approaches to using 
APIs for facilitating third-party access to healthcare databases. 


Brazil 


Background 


Among developing countries and middle powers, Brazil stands out as a pio- 
neer in the ‘informatization’ of national healthcare services. Brazil’s Unified 
Health System (SUS), a universal healthcare program established in 1988, is 
among the largest of its kind in the world.55 Since 1993, Brazil has created sev- 
eral specialized national databases pertaining to vaccinations, cancer screening 
and treatments, infectious disease surveillance, movement of restricted drugs, 
patient visits, and social security benefits. However, it has struggled to digitize 
these databases and make them interoperable across sectors and healthcare 
providers.** Consequently, private players have stepped in to build their own 
algorithms for data retrieval and linkage from these databases. Given some 
of these databases have no anonymization features,” the involvement of pri- 
vate actors has raised concerns around privacy and cybersecurity. Although an 
‘e-SUS’ platform has been in existence since 2014 to collect primary health- 
care data and population-level indicators, the development of this platform has 
been hampered by a lack of training in data-entry among healthcare workers 
as well as “bureaucratization of their work process."% Another major chal- 
lenge in digitizing and consolidating such data has been the lack of a unique 
identity program in Brazil. And finally, the absence until 2020 of an overar- 
ching data protection legislation meant there were no general legal or policy 
measures governing the handling of sensitive health data. As a result of all 
these factors, Brazil’s expansive policy infrastructure on healthcare and social 
security has historically been challenged by a skeletal digital infrastructure 
with no “semantic and technological standardization” for data.°* However, 
this scenario has changed dramatically in the aftermath of the COVID-19 
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pandemic, whose precipitation of the demand for digital healthcare services 
appears to have been seized by both government and private actors. 


The role of APIs in the digitization of healthcare 


With the onset of the COVID-19 pandemic, Brazil re-oriented the imple- 
mentation of three key policy instruments — the National Digital Health 
Strategy, 2020-2028 (NDHS), National Health Data Network (RNDS) 
(2020), and the National Artificial Intelligence Strategy, 2021 (NAIS) — to 
mitigate the spread of the coronavirus and manage the treatment of those 
infected. These policies were in advanced stages of consultations well before 
the pandemic, but Brazilian regulators were compelled by the coronavirus’ 
rapid spread to digitally unify various elements of the health system in a bid 
to address COVID-19 surveillance, immunization, adequate availability of 
hospital facilities, testing records, etc. For example, the RNDS was initially 
supposed to be rolled out as a pilot project in a single Brazilian province in 
March 2020, but was repurposed to “receive and share information [across 
the country] that could help [the government] control” the pandemic.”” Sim- 
ilarly, the national health strategy emphasized the interoperability of data 
across healthcare providers to help tackle together the spread of COVID-19.% 
Finally, the national AI strategy declared that health would be one of the first 
sectors to see the roll out of Al-driven pilot and implementation projects. 

The RNDS in particular is slated to play a critical role in the standardi- 
zation and interoperability of health data in Brazil. The NDHS declares the 
eventual objective of the RNDS to be the creation of an ecosystem where the 
“SUS, public and private healthcare organizations, technology companies, 
research centers, universities and other stakeholders share data [...] well as ex- 
ercise, test and evaluate new models, patterns, technologies and design.” In 
July 2020, Brazil made the submission of SAR S-CoV-2 diagnostic tests to the 
RNDS — whether conducted by public or private laboratories — mandatory.” 
Following this legal measure, the SUS created “accrediting systems and tech- 
nical documentation” in its ‘DATASUS’ platform to facilitate such submis- 
sion and data sharing." The technical documentation in question referred 
to a set of API standards. At the time of writing, the Brazilian Ministry of 
Health has expanded the suite of APIs available in DATASUS, and includes 
those that not only allow for the sharing of test data, but also the sharing of 
clinical studies results, immunization data, pharmacy inventories, and pri- 
mary healthcare data into RNDS.* The ministry’s Coronavirus-SUS app, 
used for contact tracing, relied on the Google/Apple Exposure Notification 
(GAEN) API developed jointly by the two companies.*° Although it remains 
possible to export data from government websites or health applications di- 
rectly, the Brazilian government has strongly encouraged the use of these 
APIs?” over other channels, creating the basis for a digital health architecture 
that is heavily reliant on middleware. 
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Al-enabled services and future plans 


The Brazilian ordinance of August 2020 that established the RNDS offers 

an insight into the role of APIs in promoting Al-driven innovation in the 

country’s health sector. The ordinance attempts to promote “interoperabil- 

ity” in:48 

a Information models, i.e., “conceptual and contextual human representa- 
tion” of data; 

b Computational models, i.e., data structures as programmed in a comput- 
ing language; and 

c “Semantic” and “syntactic” data models, i.e., human and computational 
representations respectively of “classifications, taxonomies, and ontolo- 
gies” and other information models relevant to the sector. 


From Brazil’s detailed and carefully crafted attempts to introduce standard- 
ization and interoperability in electronic health records, it is amply clear the 
country’s regulators do not see APIs simply as a quick fix towards digitizing 
the sector. Instead, Brazil’s recent national strategies on digital health and 
Al, as well as a slate of pandemic-era policies, appear to signal the creation of 
an API-centric middleware ecosystem that facilitates the sharing of personal 
and non-personal data, and in turn, promotes innovation in Al-enabled ser- 
vices. Brazil’s National Health Information and Informatics Policy (PNIIS), 
introduced in July 2021, specifically call for the use of AI to meet the needs 
of healthcare professionals and researchers." The ‘Conecte-SUS’ app, which 
provides users with a longitudinal record of their clinical history that can 
be shared with healthcare providers and researchers via DATASUS APIs, 
has already been earmarked by local governments as a DPI to promote AIl- 
enabled innovation.5 Several multistakeholder pilot projects on predictive 
COVID-19 diagnostics have already been implemented in Brazil, although it 
is unclear at the time of writing whether they have relied on APIs or single-site 
data. In any event, the ‘API-fication’ of Brazil’s digital health sector appears 
to be a deliberate and ambitious strategy to ensure public and private agencies 
can rely on large volumes of data to train and develop ML models in primary 
healthcare and diagnostics. 


Singapore 


Background 


As a middle power with outsize ambitions to shape normative and material 
outcomes in cyberspace, Singapore has long sought the comprehensive digiti- 
zation of key sectors of domestic governance. Govlech, a specialized agency 
established in 2016 to catalyze the digital transformation of Singapore’s public 
sector and user-facing services, makes a credible claim to be “the first of its 
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kind” in the world.*! Among its other responsibilities, GovTech is responsi- 
ble for the country’s “Strategic National Projects” which includes user- and 
business-facing platforms to access government services, the national digital 
identity program (Singpass), the country’s unified payment gateway (PayNow), 
and CODEX, a technology stack to standardize the development of applica- 
tions and handling of data across the country’s private and public sectors. 
While Singapore thus pioneered the development of several DPIs, it has moved 
cautiously with respect to the digitization of its healthcare sector. The fact that 
Singapore’s “worst cyber attack” implicated its national SingHealth system, 
may have been a contributing factor.” The country’s digital health policies 
initially monitored standalone products and services for quality assurance, risk 
attributes, and adverse event reporting, and it was only in 2020 that the govern- 
ment sought to address questions regarding the integrity and security of health 
data.5* As in the case of Brazil, the COVID-19 pandemic catalyzed the creation 
of legal and technical frameworks on health data in Singapore. The Regula- 
tory Guidelines on Software Medical Devices, issued in April 2020, underline 
application-layer security concerns similar to those identified by OWASP. The 
Guidelines call on software developers to ensure, among others:55 


a Use of proper authentication protocols, both at the device and API levels; 

b Development of “layered authorization models” to differentiate privilege 
levels for users and devices; 

c Encryption for data at rest and transit; and 

d Deployment of network monitoring and intrusion detection systems. 


Notably, the Guidelines also specify regulatory requirements for Al-enabled 
medical devices and services. AI/ML services that rely on ‘static’ datasets as 
well as continuous learning are required to submit descriptions of data attrib- 
utes, labels, training models, built-in audit processes, and security features, 
prior to their registration with Singapore’s Health Sciences Authority. 


The role of APIs in the digitization of healthcare 


Singapore has conceptualized its DPI as platforms, and not specific products, 
believing the latter to be an impediment to at-scale delivery of services.5€ 
Consequently, APIs have played a prominent role in connecting these 
middleware platforms to market actors and end-users. Singapore’s “platform- 
as-a-service” model is made possible by the presence of an “engagement 
layer” of APIs that connect various agencies and institutions within govern- 
ment.’ Indeed, Singapore’s (now) Chief Digital Officer has characterized 
some of these APIs as “whole-government APIs?" They allow, for instance, 
businesses to obtain licensing or regulatory approvals from multiple bureau- 
cracies through a single application. Similarly, through the integration of 
the Singpass system across various government platforms, the Singaporean 
citizen can avail of any public service through her digital ID. 
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Through the creation in 2017 ofa centralized API Exchange (APEX), Sin- 
gapore brought its “whole-government’ APIs under an umbrella framework. 
APEX allows government agencies to share data with each other as well as 
the broader public, allowing, for instance, private services to retrieve user 
data previously authenticated by the state, or citizens to submit governance 
proposals that are then channeled to the appropriate entity.” ? Given its exten- 
sive interface with government portals and sensitive data, agencies and third 
parties are required to undergo a training session and test application-layer 
security protocols before APEX onboarding is complete.” 

More pertinent to the context at hand, Singapore has also created a portal 
called ‘data.gov.sg’ that offers third parties access to public datasets through 
APIs.°! Set up in 2011, the portal was criticized in its initial years for being 
a “data dump” of files in PDF and CSV format that had to be manually 
downloaded."? In recent years, it has undergone a comprehensive transfor- 
mation, and while data files may still be downloaded, it is through APIs that 
the outside world engages with ‘data.gov.sg.’ Most importantly, the portal 
also makes available APIs that allow for the retrieval of real-time data in such 
domains as meteorology and transport. 

‘Data.gov.sg’ hosts over 100 datasets pertaining to health.°* These include 
data on infectious disease prevalence, incidence of cancer among the popu- 
lation, preventive health screening results, prevalence of so-called ‘lifestyle’ 
diseases such as hypertension, diabetes, cholesterol and obesity, hospital facil- 
ities and physicians by secondary and tertiary sectors, immunization statistics, 
and of course, COVID-19-related information. It is worth noting here that 
many health-related datasets have been made available through APIs follow- 
ing the onset of the coronavirus pandemic, although they have been in ex- 
istence for years. The rapid onboarding of health data for third party access, 
combined with Singapore’s overall vision and concerted push to promote 
“open data” governance through APIs suggests the government is heavily 
leaning on middleware-driven innovation in healthcare services. 


Al-enabled services and future plans 


In October 2021, Singapore published AI in Healthcare Guidelines (AIHGle) 
that offer non-binding recommendations to developers and adopters for the 
“safe implementation” of Al-enabled medical devices.°+ While the Guide- 
lines devote their attention mainly to questions of fairness and explainability 
of algorithmic decision-making, as well as end-user communication about 
the working of such Al-enabled devices, security considerations also figure 
prominently in the document. 

The AIHGle identifies both “data risks” and “algorithmic risks” with 
respect to the security of the Al-enabled service.” To mitigate data risks, 
the Guidelines recommend safeguards against unauthorized access (through 
APIs or otherwise) to testing, training, and clinical data. The AIHGle also 
suggests de-identifying personal data where possible, and where “individual 


The middleware dilemma of middle powers 121 


characteristics need to be retained,” using techniques such as “data masking, 
pseudonymization, or data perturbation.” To prevent re-identification, 
developers are encouraged to keep access logs and apply, where possible, 
techniques such as secure, multi-party computation. The document ac- 
knowledges algorithmic risks, i.e., security concerns pertaining to learning 
and implementation lifecycles, are more accentuated in AI/ML services that 
rely on “continuous learning” through dynamic and real-time data flows. In 
such cases, implementers are encouraged to monitor abnormal algorithmic 
behavior caused by “maliciously introduced data” or manipulations at the 
end-user level. The AIHGle places much emphasis on human intervention 
in the implementation process. Implementers of Al-enabled services should 
have “self-validation” or fail-safe mechanisms that trigger human interven- 
tion when baseline performance of the algorithm is affected and even “con- 
tingency plans” that “include shutting down the AI device and switching to 
analogue protocols.”°* 

Although the Guidelines are notable for their level of detail and specifica- 
tions with respect to cybersecurity as well as algorithmic decision-making, 
it is unclear how its non-binding recommendations will be enforced by the 
Singaporean government. The AIHGle recommends developers and imple- 
menters enter into Service Level Agreements (SLAs) that demarcate their 
respective responsibilities for the training and implementation lifecycle. 
Given the Guidelines are only a few months old at the time of writing, it is 
not clear how they apply to health data retrieved through ‘data.gov.sg,’ es- 
pecially in the case of dynamic datasets. Notably, the Singapore government 
uploaded 40 health datasets onto the portal two weeks after the AIHGle was 
published, perhaps reflecting its interest in leveraging public data to promote 
AI/ML innovation. 


India 


Background 


India has the distinction of running the largest biometrics-driven digital 
identity program in the world, which has been operational since 2009.70 The 
digital ID, called Aadhaar, is fashioned as a DPI used to authenticate the iden- 
tity of Indian citizens seeking to avail of public and private services. Aadhaar 
may be considered as the first in a suite of DPls that have since been imple- 
mented by the Indian government in sectors such as finance, logistics, and 
health. While some of these DPIs, such as DigiLocker — a cloud-based repos- 
itory where an individual may choose to store electronic records pertaining 
to identity, educational and employment history, etc. — are designed as prod- 
ucts, most middleware infrastructure built by the Indian state has taken the 
form of APIs and protocols. Examples include the Unified Payments Inter- 
face (a common railroad for instantaneous money transfers domestically), the 
Bharat Bill Payment System (an API-driven gateway for utilities payment), 
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the Goods and Services Tax Network (for collecting GST accrued to both 
the federal and local governments), etc.”! 

Since 2017, following its publication of a National Health Policy, the In- 
dian government has sought also to incubate a “federated national health 
information architecture to roll out and link systems across public and private 
healthcare providers.” 7? In 2018, India published a strategy paper on a Na- 
tional Health Stack (NHS), described as a “collection of cloud-based services” 
that run on open and interoperable APIs.” The strategy paper also mooted 
the creation of a digital health ID, a unique identifier that would allow In- 
dian citizens to not only obtain longitudinal health records from a federated 
database, but also share it with healthcare providers anywhere in the country. 
Despite its ambitious goals, however, the NHS has struggled to materialize 
on account of two reasons. The domain of health is constitutionally the pre- 
serve of state governments in India, who have been reluctant to support a 
national initiative partly on account on lack of clarity on the implications of 
the technical infrastructure for their services." Additionally, Indian regula- 
tors have also found it difficult to persuade large healthcare conglomerates to 
standardize and thereby render patient health records interoperable. As with 
Brazil and Singapore, however, the Indian government has attempted to use 
pandemic-era health surveillance powers to shift the momentum in its favor. 


The role of APIs in the digitization of healthcare 


To mitigate the spread of the coronavirus, India’s National Health Authority 
(NHA) developed and mandated the use of two platforms for contact-tracing 
and COVID-19 vaccine management. Called Aarogya Setu (‘Health Bridge’) 
and CoWIN” respectively, the development and implementation of these 
applications provided the government with the institutional fillip needed to 
create a pan-Indian technical architecture for the NHS.” The NHA has 
sought to utilize not only the personnel resources it marshalled to develop 
these applications, but also the ties built with healthcare providers to coor- 
dinate CoWIN registrations and vaccine deliveries, towards the cause of the 
health stack. The NHS is envisioned as a “building block” comprising the 


following layers:’7 


a Data layer, which includes health IDs, longitudinal Personal Health 
Records (PHR), registries of healthcare professionals and services, as 
well as other healthcare-related data such as hospital visit summaries, 
prescriptions, immunization records, etc.; 

b A protocol layer, also known as the Unified Health Interface (UHI) 
that comprises APIs enabling the seamless retrieval and sharing of data 
among various actors involved in the provision of healthcare services. 
Specifically, the UHI will comprise three categories of APIS: registry 
APIs, gateway APIs, and consent/information exchange APIs. Registry 
APIs facilitate the standardized collection and maintenance of data, 
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gateway APIs lay down specifications for access to particular healthcare 
networks, and consent APIs specify rules for “data fiduciaries,” which are 
specialized entities that manage the consent of the user to share data with 
third parties; 

c An application layer, featuring user-facing apps developed by public and 
private actors. 


As the outline above indicates, the UHI is critical to smooth functioning 
of the NHS. The API-driven layer will not only determine who can access 
sensitive health data of Indian citizens, but also the granularity of the data so 
shared with different types of entities. 


Al-enabled services and future plans 


At the time of writing, the various policies and technical specifications that 
make up the NHS are in early stages of stakeholder consultations, but the 
proposed architecture of the NHS makes it clear APIs will invariably play an 
important role in ensuring access to training data for AI/ML services. Key 
to the use of health data for training ML algorithms will be the classification 
and labeling of data, which are part of the standardized PHRs. Additionally, a 
draft policy on data retention released alongside technical specifications refers 
to the conditions under which personal data may be anonymized or pseu- 
donymized, as well as circumstances under which anonymized data should 
be deleted.” Nonetheless, the question remains as to the technical architec- 
ture that will facilitate the anonymization of data and concurrently, the use 
of training data in India’s health sector. The NHA’s blueprint for the health 
stack leaves this question open and suggests Al-enabled “clinical decision 
support systems” will be rolled out in Year 4 of its implementation.50 


APIs, AI insecurities, and middle power diplomacy 


The middleware architecture proposed or implemented by Brazil, Singapore, 
and India to digitize health data and render it available for training AI/ML 
models reveals the extent to which developing countries are reliant on APIs. 
Indeed, the three ‘models’ presented in this chapter are likely to be adopted 
by others states to jumpstart the development of Al-enabled services not only 
in health but also other sectors. States that have already made significant 
strides towards digitizing public datasets may opt, like Singapore, to make 
such data available via APIs but leave the selection of data and training of 
ML models to third parties. Those countries that have lagged behind in dig- 
itization may develop APIs to facilitate the standardized input of electronic 
records and the integration, subsequently, of national databases, as Brazil has 
done. In such cases, the respective entities responsible for digitizing health 
records may offer secondary APIs to facilitate third-party access. In yet other 
cases, states may not only standardize the creation of electronic records but 
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lay down strict policies and technical specifications — as India proposes — to 
determine how such data is shared with public and private actors alike. 

All three approaches present security concerns for AI/ML services that 
may be exploited by strategic adversaries. Developing countries that make 
datasets available for third-party use may not have the regulatory capacity of 
a small, and relatively wealthy country like Singapore to monitor or enforce 
guidelines like the AIHGle. The use of APIs available on ‘data.gov.sg’ is 
governed by Terms of Service under Singapore’s Open Data License, which 
not only restrict the use of such APIs to specific purposes but also prohibit 
downstream sub-licensing by third parties.*! The Singapore model places 
a lot of trust in self-regulation by the market. For most emerging markets, 
however, a strong cybersecurity or data protection regulator is essential to 
monitor malicious ICT activity, because an infant private sector may have 
even lesser resources than the state to mitigate them. Cyber attacks by state or 
state-sponsored actors could specifically target API vulnerabilities to manip- 
ulate or destroy information in public databases, with a view to compromis- 
ing Al-enabled services that rely on them. Additionally, poor API security 
on the part of private actors could have serious and adverse consequences for 
the integrity of the same training model data that is subsequently used by 
other developers for their respective services. Finally, the challenges of se- 
curing Al-enabled services grow in complexity when they rely on real-time 
APIs that provide ‘live’ data. At the time of writing, most datasets uploaded 
onto ‘data.gov.sg’ are static in nature. But as Singapore (and other countries) 
develop real-time APIs, regulators will need to find mechanisms that in- 
stantaneously identify and remedy serious cyber attacks on dynamic datasets, 
failing which they may cause lasting and widespread damage on Al-enabled 
services that rely on the same data. The risks associated with ML APIs and 
Man-in-the-Middle attacks have already been documented in this chapter, 
and they apply in particular to the Singapore model. 

Brazil’s API strategy is aimed at integrating electronic health records across 
the country, and making them available for governments at the federal, state, 
and local levels. As a result, AI/ML innovation in Brazil and other coun- 
tries that follow its path may be more decentralized. Municipalities and local 
healthcare providers may tie up with research institutions and market actors 
to pilot Al-enabled services in provinces by granting them access to public 
data through their own APIs. The challenge inherent in this approach lies in 
testing and auditing the security of locally developed APIs that grant third- 
party access to national databases. Local governments may not spend as much 
time and resources reviewing their APIs for security flaws as a national regu- 
lator or agency. The cybersecurity of national infrastructure is only as strong 
as its weakest link. If security considerations are not adequately baked into 
the design lifecycle of such local APIs, states will be confronted with the same 
challenges identified above with respect to AI/ML services. 

India’s approach to standardizing electronic records and specifying rules 
for their sharing via APIs may seem tightly controlled, but presents its own set 
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of problems. Government control of API design and implementation, even 
in a democracy, can result in the API development process being opaque and 
unaccountable to outside stakeholders. Admittedly, this is a problem with 
the API-fication of public databases everywhere. However, India’s wielding 
of executive power to force the adoption of DPI like Aadhaar®” and Aarogya 
Setu, and the non-responsiveness of its bureaucracy to serious security in- 
cidents8” raises the concern that a powerful government apparatus may be 
less receptive and agile to innovation. Additionally, with the state being the 
ultimate arbiter of key API decisions such as data labeling and the granularity 
of data sharing, its unaccountability vis-a-vis the research community and 
market actors can hamper investigations into security breaches of Al-enabled 
services. 

The three countries whose plans for digital healthcare have been reviewed 
here not only stand out for contrasting API-led approaches to data sharing 
with third parties. They are also influential middle powers and democracies, 
whose successes in digitalizing their economies will be closely observed by 
regional and global actors alike. As they emulate attempts by these countries 
to open up public databases to third parties, states may, in the process, also 
replicate poor cybersecurity practices with respect to APIs. 

Whatever the model, APIs are likely to emerge as vectors of cyber attacks 
on Al-enabled middleware services in middle powers. The COVID-19 
pandemic has accelerated the digital transformation of their economies and 
societies, but attendant cyber risks have also risen. If the SingHealth system 
suffered a cyber attack in 2018, API-driven infrastructure in other middle 
powers, such as India’s biometric ID system** and Brazil’s Conecte-SUS 
platform®® already suffered serious breaches during the pandemic. As gov- 
ernments rush to share data with the market through APIs for AI/ML 
innovation, they can also open a gateway for malicious actors. Handling 
as they do large volumes of information, APIs could be used to corrupt 
strategic information in databases about the demographic make-up of a 
country. Then there is the possibility of attack on the Al-enabled service 
itself. The damage caused by cyber attacks on training data and ML models 
will not only be economic, but political and psychological. Such attacks 
can erode trust in Al-enabled services among states and societies alike. 
In sensitive sectors such as health or transportation where real-time data 
is involved, cyber attacks can have catastrophic consequences, leading to 
human casualties. 

How are middle powers likely to respond in geopolitical terms to this 
middleware dilemma? 

The first possibility is that middle powers, including those states that have 
been traditionally reluctant to join alliances or plurilateral security arrange- 
ments, may seriously evaluate the possibility of collective measures to defend 
and even respond to cyber attacks on middleware infrastructure. Many mid- 
dle powers are constrained by resources to build serious offensive and cyber 
capabilities. As a result, they have “over-invested” in publicly observable 
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efforts to build institutional cyber capacity (policies, regulatory agencies, 
etc.) with little deterrent effect to show for the same.5€ Were these states 
to open their databases and middleware infrastructure to Al-enabled ser- 
vices, whose security policies as well as algorithmic models are not always 
fully transparent, they would have even less control or oversight over their 
digital networks. To detect, prevent, and mitigate cyber attacks on critical 
resources, therefore, states may seek assistance from countries with more ad- 
vanced capabilities. In particular, middle powers may enter into agreements 
with other states, including Great Powers, to protect their infrastructure from 
malicious cyber operations. A good example of collective cyber diplomacy 
in this regard is the Quad, a group consisting of the United States and three 
middle powers — Australia, India, and Japan. Motivated primarily by secu- 
rity compulsions in the wake of China’s military “assertiveness” in Asia,’ 
the Quad has committed to a number of cybersecurity initiatives, including 
a Quad Cybersecurity Partnership to share threat information about, de- 
velop software standards for, and build capacity to address cyber attacks on 
critical infrastructure. S$ Proposals for middle powers to develop collective 
diplomatic and military measures to mitigate major cyber threats are not 
new, and the ‘middleware dilemma’ identified in this chapter offers another 
compelling reason for such cooperation. 8?’ A second possibility is that middle 
powers could engage in cyber diplomacy to articulate and implement norms 
on the security of Al-enabled services. These norms, which may be articu- 
lated in intergovernmental or multistakeholder venues, may be comparable 
to guidelines on data and algorithmic risks identified by Singapore’s AIHGLe 
or address AI vulnerabilities highlighted by other prominent regulators such 
as the European Union’s ENISA.” States may also incubate or encourage 
market players to develop industry guidelines on API security, which has 
been lagging despite their growing importance to digital services, including 
Al-enabled services. 

In summary, the ‘middleware dilemma’ will nudge middle powers to play 
a more active role in cybersecurity diplomacy, with a view to ensure the 
stability of cyberspace and to enhance their own capacities to address so- 
phisticated cyber threats. Needing to sustain the digital transformation of 
key sectors, middle powers cannot afford to let discussions on AI security be 
shaped solely by Great Power politics: their proactive diplomacy could well 
lead to new norms or collective arrangements on the protection both of crit- 
ical infrastructure as well as Al-enabled services that run on them. 
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6 Artificial intelligence and 
military superiority 


How the “cyber-Al offensive-defensive 
arms race’ affects the US vision of the 
fully integrated battlefield 


Jeppe T. Jacobsen and Tobias Liebetrau 


In April 2016, then Deputy Secretary of Defense Robert O. Work pre- 
sented the Department’s latest strategic attempt to maintain US global 
military dominance, the Third Offset Strategy. The strategic direction was 
clear: the American military must better integrate the latest data-driven, 
network-based and (semi)-autonomous technologies into its operative and 
organisational processes.” The strategy's underlying assumption is that the 
more data the US military can collect, process, analyse and share across the 
organisation, the better each operative unit will function on the battlefield. 
Artificial intelligence (AI) plays a vital role in this vision — not just as the 
technology that manages the increasing amount of sensor data collected from 
civilian and military systems but also as a tool to strengthen the resiliency of 
information systems, for example against cyberattacks.> However, it is still 
unclear whether AI is going to support or undermine the fully integrated, 
network-dependent battlefield of tomorrow. 

So far, the emerging literature on the geopolitical and military implica- 
tions of AI has paid attention primarily to how strategic stability is influenced 
by the international AI competition among great powers,* or to the spe- 
cific military functions and processes that can be improved and undermined 
through the advancements in AI. A main concern in the current literature 
is that the significant investments in AI in the United States, Russia, China 
and the EU — and their respective fears of being ‘left behind’ — will result in 
an arms race.° 

This chapter makes a threefold contribution to the debate on a possible 
Al arms race. First, it argues that the risk of an AI arms race is not simply a 
matter of the undesirable framing by those political leaders and media outlets 
that do not understand the broader applications of AI.’ Rather, the chapter 
shows that the AI arms race is closely linked to the military vision of tech- 
nological superiority, which has dominated US military discourse since the 
Second Offset Strategy of the 1970s. Second, the chapter argues that the arms 
race that needs more attention does not relate simply to the general competi- 
tion among great powers in the field AI, but relates also to the specific arms 
race between offensive and defensive cyber capabilities, which is currently 
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being accelerated by AI. The current attempts to improve cyber offense and 
cyber defence with AI hold both great potential and great risk to the ability 
for militaries to dominate the information space. Building on this finding, 
the chapter takes lesson from discussions on how militaries balance offense 
and defence in cyberspace, and finally argues that Al-enhanced cyber offen- 
sive capabilities are likely to dominate, which will have a negative impact on 
the likelihood that a fully integrated, networked battlefield will ever become 
reality. 

The chapter comes in three parts. First, we situate AI in the broader US vi- 
sion of military superiority to show that the idea of an AI arms race is deeply 
rooted in US military discourse. Second, having argued that Al-enhanced 
cyber capabilities will be both a prerequisite and a challenge to the US vision 
of a future battlefield, we take stock of the current state of AI in cyber offense 
and cyber defence. And third, we discuss three dynamics that have charac- 
terised the US approach to cyber offense and defence, namely the perception 
of offensive dominance, the role of intelligence agencies, and the emerging 
market for exploits. This is done to assess the implications of the on-going 
Al-cyber arms race for the possible realisation of a fully integrated future 
battlefield. 


Imagining military superiority through 
technological ‘offsets’ 


The heavy investment in AI among great powers has increasingly been de- 
scribed in the media as an ‘AI arms race.’ Although the tendency to focus on 
killer robots borders hyperbole,’ the characterisation is not without merit if 
‘arms race’ is defined as competitive acquisition of military capability between 
two or more actors. Russia, China and the United States are all seeking to 
improve their military capability and efficiency by investing in Al innovation 
and they consider the competition a matter of who — in the famous words of 
President Putin — ‘will become the ruler of the world! Yet, scholars have 
pointed out that the emphasis on an on-going ‘AI arms race’ risks becoming a 
self-fulfilling prophesy that prompts everyone to deploy unsafe Al-systems!! 
and overlooks the many variations of technological proliferation and diffu- 
sion that must be managed very differently than if restraining an arms race. 1? 
While these points are both valid and important, the problem is not simply 
a ‘framing problem” in the media, among policy makers, and in the military. 
To understand the importance of AI dominance in the US military today, it 
is necessary to return to the US first offset strategy of the early 1950s. 

An offset strategy is an attempt to shift competitive focus in order to 
maintain an advantage over an emerging competitor despite facing restraints 
and disadvantages. President Eisenhower’s ‘New Look’ strategy from the 
early 1950s represents the first example. Facing an intensifying competition 
with the Soviet Union and declining defence budgets after WWII, Eisen- 
hower commissioned a review of the current defence policy, including an 
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exploration of alternative long term containment and deterrence strategies." 


The result was a refocus on nuclear deterrence rather than the more expen- 
sive conventional deterrence. 

After the Vietnam War, declining military expenses and increasing Warsaw 
Pact forces in Europe led US Defense Secretary Harold Brown to develop the 
Second Offset Strategy. The strategy saw new technologies as “force multipliers’ 
of combat effectiveness and as necessities for maintaining US military superi- 
ority. Asa result, the Department of Defense funded a broad range of research 
and development initiatives on military-technological innovation, primar- 
ily within intelligence, surveillance, reconnaissance (ISR), precision guided 
weapons, stealth technologies, and space-based communication. While the 
‘offset’ technologies were never tested against Soviet forces, they proved to 
be extremely efficient during the 1991 Operation Desert Storm. With the 
decisive US victory on the battlefields of Kuwait and Iraq, the Second Offset 
Strategy’s fundamental assumption that (information) technological superi- 
ority leads to military superiority became the axiom of military thinking 
in the 1990s, particularly through the offshoot-concepts of a Revolution in 
Military Affairs (RMA) and Network Centric Warfare (NCW)."* 

Based on the rapid, private-sector-driven advancements of information 
technology in society in the late 1990s, NCW came to echo the second off- 
set strategy but in the information age. Thus, information superiority was 
now the key to military superiority. NCW promised a transparent and pre- 
dictable battlefield through the continuing development and integration of 
information technological innovations into the military to increase situa- 
tional awareness, accelerate operational decision making, and get inside the 
adversary’s decision circle to dictate the pace of military operations.!° Yet, as 
Antoine Bousquet has convincingly shown, NCW — although taking as its 
point of departure the idea of a new non-linear and more complex way of 
war — insisted simultaneously on pursuing a frictionless cybernetic war ma- 
chine. Given this conceptual inconsistency as well as the inability to fulfil its 
promise of certainty, speed, and transparency in dealing with asymmetrical 
challenges in the war of Iraq and Afghanistan, the narrative of NCW lost 
some of its momentum. 

However, rather than abandoning the underlying idea of information su- 
periority as the key in warfighting, new concepts seemed only to reinforce 
the operational importance of dominating the information space. Cyber 
warfare — the ability to impact adversary computer networks for operational 
purposes — emerged as both a threat and opportunity in future wars. Relying 
on and adding to the already extensive cyber intelligence and data collec- 
tion capacities in the National Security Agency, the DoD established the 
US Cyber Command in 2009 with the intent to defend DoD information 
networks as well as to direct and conduct full-spectrum military cyberspace 
operations." 

The rapid advances in AI and autonomy in the late 2010s further ignited 
the aspiration for even more certainty, speed, precision and coordination 
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across all domains. In fact, Al and autonomy became ‘the technological sauce 
of the Third Offset Strategy’ because of US inability to match ‘tank for tank, 
plane for plane, person for person’ of the resurgent Russia’s and rising China’s 
increasing ability to fight in all domains.!* While also focusing on rethinking 
organisational and operational constructs, the Third Offset Strategy is, like 
its predecessor, primarily technological at its core. At an event at the Center 
for Strategic and International Studies in Washington D.C., Deputy Secre- 
tary Work underlined that injecting AI and autonomy into the C41 sensor 
grid of the US military would improve, for example, the ability to handle big 
data, determine patterns, provide timely relevant decision making through 
human-machine collaboration, and assist human operations through technol- 
ogy assistance such as wearables.!? 

The operational importance of AI and data as force multipliers support- 
ing future multi-domain-operations is spelled out by the Army Capabilities 
Integration Center — Future Warfare Division in a 2018 White Paper. The 
White Paper concludes that ‘artificial intelligence agents and algorithms will 
enable future force operations by processing, exploiting, and disseminat- 
ing intelligence and targeting data??? More recently, the National Security 
Commission on Artificial Intelligence in its final report accentuates the im- 
portance of AI for realising military superiority: ‘our armed forces’ competi- 
tive military-technical advantage could be lost within the next decade if they 
do not accelerate the adoption of AI across their missions. ?! 

The DoD has established the Joint Artificial Intelligence Center (JAIC) to 
ensure such acceleration in adopting AI across the US Defense. However, AI 
dominance does not only rely on the DoD. Since the Second Offset Strategy, 
the DoD perceives the ecosystem of collaboration between state, market and 
academia as the key driver of (military) technological innovation.?”? While 
the latest cyber-Al technologies developed by universities and profit-seeking 
private businesses show promising results, it comes with some risks as well as 
the last part of this section will show. 

The inability to match the conventional forces of adversaries has therefore 
led the US defence apparatus to shift competitive focus to the latest techno- 
logical developments in an attempt to maintain a military advantage. It is thus 
deeply ingrained in US military thinking that military superiority depends 
on technological superiority, which continues to rely on close collaboration 
and partnerships with the private sector. Hence, the current race to dominate 
within the field of AI is not simply a bad narrative or wrongful framing but 
part of a deep-rooted national security discourse where falling behind com- 
petitors constitute an existential threat. 

As pointed out by Paul Scharre,”* the danger of the AI arms race is that 
everyone rushes to deploy unsafe Al-systems, i.e. insufficiently tested AI 
weapon systems that can cause devastating unintended consequences. How- 
ever, even thoroughly tested AI systems bear risks. The data-driven, net- 
worked, Al-infused, multi-domain US regime of warfighting imagined 
in the Third Offset Strategy comes indeed with its own paradox. The recent 
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attempt to achieve information superiority has resulted in a larger number of 
complex systems and system of systems. Such interconnected systems — while 
potentially improving military efficiency and capability — simultaneously 
broaden the attack surface and increase the vulnerability to cyberattacks.2* 
Here, AI becomes a Janus-faced phenomenon as it promises to improve both 
network defence and network intrusion capabilities. The more pertinent Al 
arms race that will determine the future of the integrated battlefield is hence 
the one that currently takes place between offensive and defensive cyber ca- 
pabilities. Before we go on to discuss how this arms race is likely to develop 
in the near term, it is necessary to briefly take stock of how AI is currently 
being developed and deployed to strengthen and streamline cyber defence 
and offensive techniques. 


AI in cyber defence and offense — where are we now? 


At its core, cyber defence is about knowing your network better than the 
adversary.”> The fundamental challenge is, however, that an institution like 
the US Department of Defense has more than 15,000 networks and many 
million attached devices that need updates from time to time.” Such a task, 
when done manually, is time consuming and it is also near-impossible to 
make sure all systems work only as intended. It is, thus, no surprise that the 
research agency of the US Defense, DARPA, is heading efforts to automate 
software that identifies and patches vulnerabilities in IT-systems. Humans 
must be taken out of loop, as former head of NSA, General (ret.) Keith Alex- 
ander emphasised in a Senate hearing.” 

An illustrative and much cited example of the attempt to take humans out 
of the loop of cyber defence was DAR PA's 2016 Cyber Grand Challenge — 
the agency’s attempt to get companies and universities to compete against 
each other to develop innovative AI solutions that could detect intrusions 
and identify, patch and exploit vulnerabilities. The winner system, Mayhem 
did not only win the first price of $2 million dollars, it also won a contract 
with the US Defense.”* Mayhem was particularly innovative in its ability 
to balance performance and security when rolling out patches as well as 
in its ability to proactively make it more difficult to exploit vulnerabilities 
even before they are identified.”” However, despite these improvements, 
Mayhem lost a similar ‘capture-the-flag’ competition at the hacker confer- 
ence DEFCON against human-machine teams.*” And while we have seen 
several example of AI systems consistently beating humans in games like 
Chess and Go or in military flight simulations exercises,’! we still need 
to see successful examples of operative ‘self-healing’ IT-systems in use in 
the defence networks. In contrast, automated Intrusion Detection Systems 
have been implemented in the US Defense. The NSA-developed Sharkseer- 
programme uses different forms of AI techniques to conduct incoming 
traffic and e-mail inspections to identify zero-day exploits from the most 
advances threat actors.°” 
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Yet, Sharkseer and the systems competing at the Cyber Grand Challenge 
are mainly built on rule-based AI and only to a lesser degree machine learn- 
ing.” The publicly available cyber security solutions that are based on artifi- 
cial neural networks and are trained through machine learning processes are 
currently mainly being developed in the private sector. The security com- 
pany Fortinet, for example, has spent the last seven years developing a detec- 
tion system based on supervised, unsupervised and reinforcement learning.” 
The hope is that the detection system will be able to identify and block 
malware in real time. 

The potential of AI in cyber defence does not only relate to the identi- 
fication and blocking of malware. A key global cyber security challenge is 
the reuse of code. DARPA and US Air Force Research Laboratory are 
currently financing a project that seeks to develop a system that uses machine 
learning to evaluate the quality of source code and thereby help programmers 
reduce the risk of vulnerable software.’ Some projects go even further. Rice 
University’s Bayou-project — also financed by the US Defense — and projects 
at Cambridge University and Google are currently developing Al-systems 
that recognise and evaluate the intent of programmers, ultimately to avoid 
human typos and help streamline otherwise very complex coding.” 

Despite several promising projects and large progress in the general de- 
velopment of AI, General Alexander’s vision to get humans out of the cyber 
security loop still lies in the future. Al-systems still needs human assistance 
and few network administrators feel comfortable letting Al-systems inde- 
pendently re-programme code in critical, active environments, for example 
in the military. The ‘Master AI’ that runs the whole network, has access to 
billions of actions on a million devices and links data from all enterprise soft- 
ware applications and management systems, is still fiction. 

Yet, it remains vital for the resiliency of the future integrated, network- 
based battlefield that an Al-enhanced cybersecurity system in milliseconds 
independently can quarantine suspicious activity, adapt firewalls and authen- 
tications, and isolate parts of the network.>® Here, the more or less rational 
worry of network administrators are not the only obstacle. Machine learn- 
ing algorithms need large quantities of data in a cloud where all data from 
all systems are stored in the right format — and not in a variety of formats 
across countless servers as is currently the case in most enterprises.°” But 
even though, the technology to create a Master Al-system will most likely 
be here before we expect it, the will to actually let such a system run on an 
active critical network, knowing that mistakes and unintended consequences 
are inevitable and even part of the systems learning process, is probably not 
as imminent. 

The reluctance is further fuelled by the fact that AI is not only relevant 
for cyber defence. As mentioned already, it has offensive potential as well. 
The company behind Mayhem, for example, has already developed and con- 
tinues to experiment with different techniques aiming to undermine Al 
systems — either through identification and exploitation of vulnerabilities 
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in AI algorithms or through feeding adversary AI software with erroneous 
training data.*° The rest of this section turns to the current development in 
Al-enhanced cyber offense. 

The use of offensive cyber fuelled by AI is still at an early stage, but ‘cyber 
attacks augmented by AI portend tailoring and manipulating the human side 
of important societal systems as well as introducing the risk that comes from 
moving technical skill from the hacker to an algorithm | At the same time, 
offensive and defensive cyber capabilities tend to develop together, as it is 
hard to safeguard the appropriate defence measures without knowledge of 
how state-of-the-art malicious cyber operations are executed. 

Al and machine learning have already been used to improve the content of 
phishing e-mails, avoid spam-filters, and better map and systematise the col- 
lection of data on specific targets, which continues to be the foundation for 
most cyberattacks. The open source, neural network SNAP_R is an illustra- 
tive example of a very successful software that is able to map data across social 
media networks and send targeted phishing links.*? Other current uses of AI 
to improve cyber offense relate to the attempt to determine the success rate 
of a cyberattack, to identify vulnerabilities in various software and networks, 
and to re-programme malware to avoid anti-virus." 

Most interestingly, AI can be used maliciously by integrating it with 
malware.** A myriad of options exist for malware developers to utilise AI. 
Simply put, malware will contain a definition of what purpose it is meant 
to serve and who it is intended for. The purpose may be to create two-way 
communication so that the attacker can copy data or encrypt files. Who the 
malware is intended for is also crucial. Should the malware simply compro- 
mise a specific user, or should it try to identify other target persons of the 
compromised user, for example by extracting data from contacts stored in the 
mail client, which can be used to escalate the attack. If purpose and goals are 
obscured, it is difficult to detect the malware. 

At the Black Hat USA 2018 conference, IBM researchers presented ‘a 
new breed of highly targeted and evasive attack tools powered by AT called 
Deeplocker.? The malware ‘conceals its intent until it reaches a specific 
victim’ and ‘it unleashes its malicious action as soon as the AI model iden- 
tifies the target through indicators like facial recognition, geolocation and 
voice recognition.”*° What made the Deeplocker malware extraordinary was 
the complex nature of the neural network that allowed it to conceal infor- 
mation about its target and the purpose deep in the code (hence the name). 
Moreover, Yu et al. underlines that ‘even if the malware had been found, it 
would be very difficult for malware analysts to determine who or what the 
malware was searching for. Without knowing these things, it would be im- 
possible to decipher the trigger condition, meaning that the payload would 
never be unlocked and remain unable to be studied.”*” 

Deeplocker and most of the other Al-infused techniques used for cyber of- 
fensive purposes are currently being developed in the private sector or as open 
source products. There is scarce amount of unclassified information on how 
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the DoD is current integrating AI into its offensive cyber capabilities in the 
US Cyber Command or the NSA. Yet, with the strategic focus in the DoD on 
maintaining technology superiority through collaboration with the private 
sector, it is very likely that the US military and intelligence agencies remain 
at the forefront of integrating AI into their cyber offensive capabilities. 

This section has shown that AI provides great opportunities to revolu- 
tionise both defensive and offensive cyber capabilities. The development of 
preventive cyber AI — used to detect malware and prevent malicious cyber 
operations — is thus taking place in competition with the development of 
more sophisticated Al-integrated malware. The interactive symbiosis with 
escalating competition between the use of AI for defensive and offensive 
purposes can best be described as an AI arms race in cyberspace — a race that 
does not seem to stop for the time being. 

But where are we heading? The question is vital as the United States is 
likely to continue its pursuit of its military vision of even more networked 
warfighting capabilities powered by integrated ICT as the key to military 
superiority. The next section takes lessons from the race between offensive 
and defensive cyber capabilities that has been on-going for more than two 
decades. If AI is simply tapping into and enhancing these existing dynamics, 
then the balance between offense and defence in cyberspace is instructive of 
where we are heading. 


The cyber-AI arms race — lessons from balancing offense 
and defence in cyberspace 


Jacquelyn Schneider has identified a capability/vulnerability paradox in mod- 
ern digital-enabled warfare: networked technologies promise to improve mil- 
itary efficiency and capability but simultaneously broaden the attack surface, 
thus increasing the vulnerability to cyberattacks.** The trade-off between 
military efficiency and vulnerability mitigation depends on the balance 
between cyber defensive and offensive Al-enhanced capabilities. In other 
words, the cyber AI arms race is likely to determine whether or not the very 
efficient, fully integrated networked battlefield becomes reality. In the final 
part of the chapter, we introduce three dynamics that characterise the existing 
US approach to cyber offense and cyber defence: (1) the perception that cyber 
offense dominates cyber defence, (2) the dominant position of intelligence 
agencies in cyberspace, and (3) the reliance of the market for exploits. Each 
of the three is discussed with the view to how they are likely to influence 
the cyber-AI arms race and thus ultimately the US vision of fully integrated 
battlefield. 


Cyber offensive dominance: technologically or socially determined? 


Traditionally, we have experienced a consensus among US politicians and 
military that cyber security is dominated by the offence.*? This has been 
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tied to the idea that cyberspace fundamentally changes international com- 
petition and warfare. As early as 1991, the National Academy of Sciences 
opened a report stating, “We are at risk. Increasingly, America depends on 
computers. Tomorrow’s terrorist may be able to do more damage with a 
keyboard than with a bomb.” At the same time, the idea was born that 
readily-attainable cyber weapons would allow rouge states or individuals to 
cause massive destruction and the term electronic (later cyber) Pearl Harbor 
was coined. The endurance of both the idea and the term was depicted in 
2012, when then-Secretary of Defense Leon Panetta became famous for say- 
ing that the United States faced the threat of cyber Pearl Harbor and a cy- 
berattack perpetrated by nation states or violent extremists groups could be 
as destructive as the terrorists attack on 9/11.5! At a scholarly and analytical 
level, the alleged systemic offensive advantage pertains to arguments regard- 
ing the design of the internet, endless amount of software vulnerabilities, the 
attribution problem and low barriers of entry.” Combined with the fact that 
securing networks are both difficult and incredibly resource intensive, then 
US Cyber Commander Michael Rogers argued that the United States must 
think about how to increase its capacity on the offensive side to get to that 
point of deterrence.*? 

In recent years, scholars have questioned the dominance of offensive 
cyber.5* They offer different understandings of the role of technology and 
present different types of obstacles for the success of offensive cyber ca- 
pabilities, including organisational, economic and knowledge/intelligence 
barriers. Rebecca Slayton argues, that the sources of cyber offensive or 
defensive advantage are not determined by technology alone. Instead of 
relying on technological determinism, she suggests to study the ‘organi- 
zational processes that govern interactions between technology and skilled 
actors-processes such as software updating, vulnerability scanning, and ac- 
cess management.55 Slayton proposes that the apparent success of the offense 
stems from “poor management and the relatively limited goals of offense, 
rather than a technologically determined offensive advantage.’°° Moreover, 
she asserts that one can only ‘assess the offense-balance of cyber operations 
between two adversaries, but not of cyberspace,’ because the balance is dy- 
adic and not a systemic variable.57 

Despite these scholarly insights, it has proven difficult to change the per- 
ception of offensive dominance in cyberspace among both practitioners and 
policy makers. As a result, US strategic documents continue to emphasise 
the insufficiency of relying on purely defensive measure when preventing 
and managing the cyberattacks.°* In fact, the recent scholarly and political 
interest in the various ways in which cyberspace has reinforced a new form 
of strategic competition in the ambiguous space between war and peace has 
only made the US military’s exploitation of vulnerabilities and hacking of 
foreign networks appear more relevant both for offensive and defensive pur- 
poses.” The US Cyber Commander Paul Nakasone emphasised that only by 
executing operations outside of US military networks — the so-called, defend 
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forward missions — is it possible for the United States to proactively defend 
and compete with adversaries in cyberspace.09 

The perceived need to dominate the cyber offensive space will likely create 
incentive to integrate AI into these capabilities. Specifically, this means that 
the US military is likely to prioritise the development of Al-enhanced cyber 
capabilities that can improve the identification of software vulnerabilities, 
the targeting of cyberattacks, and the hiding of the payloads. However, the 
last decades of cyber capability development have taught us that capabilities 
are difficult to retain: Exploits are often dissected after they are used, the 
US military and intelligence community have not always been able to keep 
their capabilities secret, and the more capable adversaries have proven able 
to develop their own sophisticated capabilities. All these elements lead us 
back to the observed paradox that the US military goal of achieving infor- 
mation superiority through increased development and deployment of ICT, 
particularly AI, is likely to enhance the country’s military capability, while 
simultaneously making it “extremely vulnerable because of increasing de- 
pendencies on information.! The increase in interconnected, ICT-enabled, 
and Al-enhanced systems — while promising to improve military efficiency 
and capability — simultaneously broadens the attack surface and increased the 
vulnerability to cyberattacks.62 


The golden age of intelligence agencies: will they remain the most 
powerful in cyberspace? 


While the debate about AI and cyber conflict often pivots on issues of mil- 
itary strategy and armed conflict, computers and networked systems have 
historically been built and used at the cutting edge primarily by the intelli- 
gence communities. The emergence of the Internet and networked comput- 
ing thus provided intelligence agencies with an unprecedented opportunity 
for espionage and surveillance.°° In the new digital environment, the same 
often flawed software and hardware have been used globally by governments, 
terrorists and criminals alike. When the amount of data exchanged simul- 
taneously increased exponentially throughout the 1990s and 2000s, it is no 
surprise that intelligence agencies throughout the period continued to invest 
heavily in improving the collection capabilities in this new domain — and col- 
lected as much data as possible. As a result, the intelligence agencies became 
the most capable players in cyberspace — and have been ever since. However, 
as a highly digitised society, the United States was deeply dependent on the 
same vulnerable IT-systems that the NSA and other US intelligence agencies 
were exploiting. As Ben Buchanan puts it, ‘the means of secret stealing are in 
tension with the means of secret security. 6’ 

NSA’s initial answer to this paradox between collection and defence was 
the invention of the notion, NOBUS (Nobody But Us). Here, the prem- 
ise was that the United States would seek to secure its networks and com- 
munication against all collection techniques except when these techniques 
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were deemed to be so complex that only the United States would be able 
to develop and use them.°° The difficult judgments on other actors’ cyber 
capabilities vis-a-vis American capabilities seemed to work initially. How- 
ever, the Snowden revelations as well as cyber incidents such as Heartbleed 
and WannaCry increased the public demand for a more transparent process 
through which intelligence services and other public agencies judge whether 
an exploitable software vulnerability should be fixed for defence purpose or 
whether it should be retained for offensive or espionage purposes. 

Until 2010 there was no process for sharing cyber threat intelligence be- 
tween agencies or for working out the various equities between offensive and 
defensive mandates, but the US White House Cybersecurity Coordinators 
under President Obama and President Trump both responded to the public 
criticism by clarifying the official cross-institutional process through which 
such judgements were made. The process is known as the Vulnerabilities 
Equities Process (VEP).” If an agency wants to keep a zero day, it has to 
argue its case through the VEP to an Equities Review Board chaired by the 
National Security Council (NSC) and attended by representatives from other 
public agencies, including those most concerned with the security of critical 
U.S. infrastructure like the Department of Homeland Security (DHS) and 
the Department of Commerce. 

Following the 2017 global cyber incidents WannaCry and NotPetya, the 
VEP made headlines. One of the reasons why WannaCry and NotPetya 
spread so quickly, globally and massively was because attackers used an ex- 
ploit developed by — and later stolen from — the NSA (known as EternalBlue). 
Consequently, the US government and the NSA were criticised for their con- 
tinued practice of stockpiling vulnerabilities for later use and leaving citizen 
and industry users vulnerable.°® Moreover, critics have emphasised that even 
after the 2017 improvement to the process, it still contains open-ended lan- 
guage of the “exception to disclosure.’ In other words, the decision on when 
to disclose and when to retrain IT vulnerabilities that can be used offensively 
still depends on the internal power hierarchy among the entities working 
with cybersecurity, cyber espionage and cyber warfare in the United States — 
a hierarchy that has NSA at the top. This has led Knake and Schwartz to 
recommend a transfer of the role of VEP executive secretary functions from 
the NSA to the more defensively minded DHS, an increase in transparency 
through an annual public report and a strengthened independent oversight.” 

As these recommendations have not been implemented, the NSA’s prefer- 
ence for intelligence collection continues to dominate — even if sometimes at 
the expense of individual cyber security.’' The agency is therefore likely to 
continue its exploration of the various ways in which to improve its exploita- 
tion techniques with AI. The US adoption of the cyber doctrine of persistent 
engagement and defend forward will only strengthen the need for intelli- 
gence agencies. The NSA’s development of Al-enabled cyber capabilities 
that are even more effective in identifying and exploiting IT vulnerabilities 
will then reinforce the long-standing debate on whether to retain zero day 
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vulnerabilities for intelligence purposes, use them in military operations or 
disclose them to vendors so they can be patched. In short, if the institutional 
power dynamics in the United States remain unchanged, then the dominant 
norm in cyberspace is likely to be an intelligence norm, where intelligence 
agencies seek to integrate AI into their exploitation capabilities. This would 
jeopardise the US military goal of achieving information superiority through 
increased development and deployment of AI. 


The market for exploits: can the private sector be tamed? 


ICT technologies and infrastructures are primarily developed, owned, oper- 
ated and controlled by private companies and cyberspace is often depicted as a 
non-state-centric environment.’ Unsurprisingly then, as computers became 
the primary tool and target of espionage and crime during the 2000s, a mar- 
ket developed around cyber security services. A part of this market relied on 
individual hackers searching for vulnerabilities and selling them to software 
companies or specialised cyber security companies. The latter could then use 
the vulnerabilities to simulate a cyberattack on a system, and ultimately help 
the companies in improving their intrusion detection software. However, as 
demonstrated by Nicole Perlroth these ‘defence’ dynamic was outcompeted 
by government agencies who could pay more.’? Perlroth tells the story of the 
cyber security company iDefense that in the early 2000s started to receive 
calls from various government entities willing to pay $150,000 for a bug iDe- 
fense was buying from individual hackers for $400.” 

The VEP described above is thus only a small part of a far oo ecosystem 
of vulnerability and disclosure. According to Jason Healey,”° the ecosystem 
‘includes security researchers who find new vulnerabilities, vendors who 
patch them and perhaps seek them out through a corporate or independent 
“bug bounty” program, grey markets and other intermediaries who help bro- 
ker connecting researchers to vendors (to patch) or attackers (to gain illicit 
entry), and government agencies that are sometimes attackers and sometimes 
defenders.’ Today, governmental agencies regularly acquire cyber tools, 
including a wide range of cyber weapons, from private actors. The zero- 
days markets trade in vulnerabilities and exploits, ‘the relative proportion of 
which differ according to whether a market is characterised as white, grey 
or black"? US government agencies are said to be the main purchasers of 
vulnerabilities and exploits on the grey market.’’ One of the most discussed 
examples is the 2016 unlocking of the iPhone used by a terrorist in the San 
Bernardino shooting. The phone was unlocked by a small Australian hacking 
firm, ending a historic standoff between the U.S. government and Apple.”® 

As of 2021, there is no international agreement on if or how cyber weap- 
ons should/can be regulated, nor does any such regime seem to be pending. 
According to Tim Stevens,” the lack of a global governance regime for cyber 
weapons can be explained by the constraining power of the Tallinn Manual, 
US involvement in markets for cyber weapons, the institutional power of 
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internet technologies, and diplomatic claims to sovereignty that mask op- 
erations of compulsory power. Yet, there are attempts to build international 
import-export control agreements. The most famous — The Wassenaar Ar- 
rangement on Export Controls for Conventional Arms and Dual-Use Goods 
and Technologies — is a multilateral arms control treaty between 42 countries 
that has increasingly seen the addition of software to the list of prohibited 
trade items. Initially focused on traditional weapons, war and unique prod- 
ucts like chemicals, the Wassenaar Agreement has been updated in recent 
years to include software that encompasses intrusion command-and-control 
features. 

However, the US and private tech companies worked actively to limit the 
scope of the controls, as they feared overly-broad controls limiting research- 
ers ability to identify and correct security vulnerabilities and criminalising 
essential tools for stopping malware. Without further and firm support of the 
United States, it is unlikely that voluntary international regulation mecha- 
nisms like the Wassenaar Agreement will achieve its desired effect and proper 
restrictions be put in place. On the contrary, Stevens argues that the US’ po- 
sition as ‘the dominant producer and consumer of cyber weapon components 
and research disincentives market regulation and encourages international 
trade in code entities like zero-day exploits.’°° A global governance frame- 
work for cyber weapons is only emerging hesitantly. As the development of 
an effective architecture for regulating and prohibiting weapons is generally 
a slow process,”! the prospect of governance of AI fuelled offensive cyber 
capabilities is discouraging. 

In sum, the continued domination of intelligence thinking, the grey mar- 
ket trading by government agencies purchasing cyber weapon components, 
and the lack of import-export restrictions on cyber weapons all suggest that 
the cyber-arms race is likely to continue — also with the addition of AI. This 
risk further undermining the long-term stability and security of/in cyber- 
space for numerous reasons. It increases the general risk of accidents and inci- 
dents, it creates a strong incentive for research to be aimed at writing exploits 
rather than detecting and reporting vulnerabilities, it does not encourage 
software vendors to internalise their security costs, it crowds out defensive, 
resilience and stability efforts stimulated by other government agencies, and 
it contributes further to the strategic predicament of an inevitable cyber AI 
arms race. 


Conclusion: the cyber Al-arms race is deep rooted but 
not inevitable 


The chapter paints a bleak picture: The current race to dominate within 
the military field of AI is not simply a bad narrative or wrongful fram- 
ing that can be easily replaced but is part of a deep-rooted national se- 
curity discourse where falling behind competitors constitute an existential 
threat. The chapter also pointed to several reasons why investment in Al for 
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cyber-offensive purposes is likely to continue and dominate cyber-defence 
Al. Such offensive-skewed cyber-Al arms race risks undermining the US 
vision of military superiority through a fully-integrated, networked battle- 
field: Either adversaries develop Al-infused tools to undermine the integrity 
and accessibility of US battle networks or commanders and decision makers 
develop a lack of trust in the operability of these networks. As a result, Al 
could lose its appeal as the object that promises military superiority, and a 
new technology will emerge that promises battlefield dominance when fully 
developed and integrated sometimes in the future. 

A conclusion about an inevitable arms race, however, is not written in 
stone. Both the scholarly and practical debates on cyber offense and cyber de- 
fence in the United States offer alternative paths. Slayton has already shown 
how particular organisational and management cultures are reproducing 
cyber offense as the dominant perspective, rather than being technologi- 
cally determined.5? Knake and Schwartz have introduced a range of policy 
recommendations that seek to strengthen the role of cyber defensive and 
cybersecurity apparatus in the United States vis-a-vis the intelligence agen- 
cies and the US Cyber Command.** Lastly, the attempts to promote a cyber 
export control regime — inspired e.g. by the Wassenaar Agreement — seek 
to weaken a private market for exploits that reproduce offensive dominance 
in the cyber domain. Each of these debates, if translated into policy, would 
strengthen the private businesses that work to strengthen the defensive side 
of the offensive-defensive arms race through new innovative AI solutions. 
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Introduction 


In information societies, maintaining a technological advantage is pivotal to 
the success of national defence and security measures. This is why over the 
past two decades there have been growing efforts to design, develop, and 
deploy digital technologies in this domain. Efforts span from the internet of 
things (IoT) to robotics and artificial intelligence (AI). 

AI, in particular, has shown to have great potential to aid national defence 
and security practices. This technology has become efficient and effective in 
addressing complex and important tasks; both within the civil and military 
domain. Indeed, scholars, policy-makers and military experts observe that 
there is an on-going global race for the development of AI as a defence and 
security capability. For example, the latest national defence and innovation 
strategies of several governments — UK,” US,’ Chinese,* Singapore,” Japa- 
nese,° and Australian’ — explicitly mention AI capabilities, which are already 
deployed to improve the security of critical national infrastructures, such as 
transport, hospitals, energy and water supply. 

The possible applications of AI in national defence and security are virtu- 
ally unlimited, ranging from support to logistics and transportation systems 
to target recognition, combat simulation, training, and threat monitoring. 
This potential is coupled with serious ethical challenges. If left unaddressed, 
these challenges could hinder the adoption of AI for national defence and 
security or pose significant problems for our societies, like escalation of con- 
flicts, the promotion of mass surveillance measures, as well as the spreading 
of misinformation or breaches of individual rights.® However, these ethical 
challenges are serious and hard to overcome but they can be addressed suc- 
cessfully,” if the design, development, and use of AI are informed by ethical 
considerations and guidance. 

The goal of this document is to offer guidance by identifying ethical prin- 
ciples to inform the design, development, and use of AI for defence and 
security purposes. 

These principles should not be taken as an alternative to national and in- 
ternational laws; rather they offer guidance to the use of AI in the defence 
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and security domain in ways that are coherent with existing regulations. In 
this sense, these principles indicate what ought to be done or not to be done 
“over and above the existing regulation, not against it, or despite its scope, or 
to change it, or to by-pass it (e.g. in terms of selfregulation).”!” 

As we shall see in the section “Ethical Guidelines for the Use of AI”, 
ethical guidelines for the use of AI are often designed to be coherent with 
values of the organisation or key constitutional values of the country issu- 
ing the guidelines. For example, the principles defined by the US Defence 
Innovation Board (DIB) rest on International Humanitarian Law, as well as 
on core values of the US Armed Forces.!! The European Group on Ethics 
in Science and New Technologies points towards EU Treaties and the EU 
Charter of Fundamental Rights as a starting point for the development of 
ethical values.! 

In the rest of this report, the second section describes the methodology 
used for this analysis. The third section offers a definition of AI and an anal- 
ysis of the ethical problems linked to current uses of this technology for de- 
fence and security. The fourth section introduces five principles to guide the 
specific deployment of AI for national defence and security. The fifth section 
concludes the chapter. 


Methodology 


The first step to identifying viable ethical principles to guide the use of AI for 
national defence and security is the identification of the ethical problems that 
this use may pose and that the principles should address. However, the choice 
as to how to identify these problems is not a trivial one. One may think of 
developing a complete taxonomy of ethical issues of AI in defence and secu- 
rity; but this is unfeasible and of little value: the taxonomy would be quickly 
outdated by the rapid developments in AI and its application to new uses. At 
the same time, different ethical problems may become evident when consid- 
ering Al from different points of view. For example, some ethical problems of 
AI are inherent to the design and development processes, others emerge with 
the specific domain and purpose of deployment. Hence, the choice of level of 
analysis (or levels of abstraction) becomes crucial. Analyses that disregard the 
specific domain and purpose of deployment risk defining ethical principles 
which are too generic to provide any concrete guidance. At the same time, 
analyses that try to address all possible ethical challenges related to the use 
of Al in a specific domain risk losing sight of the need to harmonise ethical 
principles for uses of Al in one domain with the broader set of values under- 
pinning our societies. 

To avoid both these risks, our analysis relied on the method of the Levels of 
Abstraction (LoAs)!5 to identify the ethical problems related to the use of AI in 
defence and security. Before delving into our analysis, let us introduce LoAs. 

LoAs are used in Systems Engineering and Computer Science to design 
models ofa given system." They are also widely used in Digital Ethics and have 
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been applied in this field of research to address several key issues, like identi- 
fying the responsibilities of online service providers,° offering guidance on 
the deployment of tracing and tracking technologies during the COVID-19 
pandemic,'© analysing the possibilities of deterrence in cyberspace," or con- 
sidering the ethical implications of trust in digital technologies.’ 

The method starts from the assumption that any system can be observed 
by focusing on specific properties while disregarding others. The choice of 
these properties, i.e. the observables, depends on the observer’s aim. For ex- 
ample, for an engineer interested in maximising the aerodynamics of a car, 
the observables may be the shape of its parts, their weight and the materials. 
For a customer interested in the aesthetics of the car, the observables may be 
instead its colour, the car’s interiors, and the overall look. The engineer and 
the customer observe the same car (system) at different LoAs, which will en- 
able them to define different models of the car. 

Thus, a LoA is defined as a finite but non-empty set of observables accom- 
panied by a statement of what feature of the system under consideration such 
a LoA stands for. It is important to stress that a LoA does not reduce a car 
to merely the aerodynamics of its parts or to its overall look. Rather, a LoA 
is a tool that helps to make explicit the system observation perspective and 
constrain it only to those elements that are functional in a particular observa- 
tion for the chosen aim.!? LoAs can be organised in a gradient of abstractions 
(GoA), this is a way to consider a range of LoAs. A LoA can have a lower or 
higher granularity. 


The quantity of information in a model varies with the LoA: a lower 
LoA, of greater resolution or finer granularity, produces a model that 
contains more information than a model produced at a higher, or more 
abstract, LoA.20 


When considering the ethical challenges of AI used in national defence and 
security one may focus on different LoAs, for example one may decide to con- 
sider only ethical problems emerging during the design stage of this technol- 
ogy and disregard the development and deployment steps. Similarly, ethical 
analyses may focus only on the intention of use or only on the effects of use 
of AI in this domain. Given the goal of this document, we choose a GoA that 
combines two LoAs: LoApurpose and LoAgthics. The observables of LoApurpose 
are the immediate purposes (henceforth: purpose) of deployment of AI. The 
observables of LoAgthics are, for any given purpose, the aspects of the design, 
development and deployment of AI that may lead to un/ethical consequences. 

It is worth stressing that the purpose is not the function of a specific tech- 
nological artefact, an artefact with the same function may pose different eth- 
ical problems when deployed for different purposes. For example, consider a 
hypothetical AI image recognition system working on different databases. In 
one implementation the system may be used to grant access to a facility, while 
in other it may be deployed to identify targets of the battlefield. While the 
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function — recognising images — of the system remains the same, the purpose 
changes and with it the ethical implications to consider. The choice to focus 
on purposes of use rather than on the function of the technology rests on two 
reasons: the malleability of digital technologies and the goal of the analysis that 
we provide in this chapter. Malleability refers to the fact that digital technol- 
ogies, even the most sophisticated, can easily be repurposed. As Moor put it: 


[Digital technologies] can be shaped and molded to do any activity that 
can be characterized in terms of inputs, outputs, and connecting logical 
operations. Logical operations are the precisely defined steps which take 
a computer from one state to the next. The logic of [digital technologies] 
can be massaged and shaped in endless ways through changes in hard- 
ware and software.”! 


Because of their malleability, un/ethical implications of digital technologies, 
Al in particular, are not necessarily defined by their design function as much 
as they are determined by the purpose with which these technologies are de- 
ployed. Within the defence and security domain these purposes can be clearly 
identified and are likely to shape both current and future uses of AI, thus they 
can inform the identification of ethical implications of the current and future 
uses of AI in this domain. 

The goal of this analysis is not to define a comprehensive taxonomy of 
AI technologies and their related ethical implications, but to offer criteria to 
identify the ethical challenges linked to use of AI in the defence and security 
domain and provide ethical guidance to address them, this is why the focus 
on the purpose of the use is key. In this sense, the reader should consider the 
rest of this document not so much as a map of the possible uses and related 
ethical implications, but as a compass to orient practitioners and researchers 
working on the ethics of AI for national defence and security. 

The LoAs embraced for this analysis have a medium granularity, as they 
focus on specific purpose of deployment in the domain of defence and secu- 
rity. Thus, they identify problems (and inform the definition of principles) 
that are not directly applicable to other domains, e.g. healthcare or public 
policy. At the same time, the LoAs abstract from specific contexts (e.g. naval 
or aviation) of AI deployment within the defence and security domain and 
disregard the variation of ethical challenges that may occur between different 
contexts. Consider, for example, the different problems and related solutions 
for using AI to aid submarines or aviation operations. 

Using this GoA we identified three purposes of deployment of AI in de- 
fence and security: sustainment and support, adversarial and non-kinetic, 
adversarial and kinetic. We shall delve into these categories in the next sec- 
tion, but let us describe them briefly here. Sustainment and support uses of 
AI refer to all cases in which AI is deployed with the purpose to support 
‘back-office’ functions, as well as logistics distribution of resources. This cat- 
egory also includes uses of AI to improve the security of infrastructures and 
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Al for national 
contingency operations 


Figure 7.1 The three purposes of use of AI for national defence and security. 


Source: Author’s creation. 


communication systems underpinning national defence and security services. 
Adversarial and non-kinetic uses of AI range from uses of AI to counter 
cyber-attacks to active cyber defence, and aggressive cyber operations with 
non-kinetic aims. Adversarial and kinetic uses refer to the integration of AI 
systems in combat operations, these range from the use of AI systems for 
threat identification to lethal autonomous weapons systems (LAWS). 

The ethical principles for the use of AI in the defence and security domain 
that we provide in the fifth section refer to the purposes of use defined in this 
section. It should be noted that, the principles defined in this chapter focus 
only on sustainment and support uses and on adversarial and non-kinetic 
uses. Adversarial and kinetic uses will be the focus of further work, which 
will build on the findings of this study. 


Ethical challenges of AI for defence and security 
purposes 


AI draws upon a variety of approaches, methods, and models for use across 
a broad range of purposes. For the goal of this chapter, however, we can ab- 
stract from specific technical aspects (we can disregard, for example, whether 
the system under analysis is a statistical or a subsymbolic one) to focus only on 
the features of AI systems from where ethical challenges arise. At this LoA, 
Al is described as 


a growing resource of interactive, autonomous, and self-learning agency, 
which can be used to perform tasks that would otherwise require human 
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intelligence to be executed successfully. This definition identifies in 
AI applications a growing resource of interactive, autonomous, and 
self-learning agency, to deal with tasks that would otherwise require hu- 
man intelligence and intervention to be performed successfully. ”? 


This combination of autonomy and learning skills underpins both beneficial 
and ethically problematic uses of AI. When considering the latter, at a high 
LoA, five key challenges have been identified in the relevant literature.” 
These are: 


e enabling human wrongdoing; 

e reducing human control; 

e removing human responsibility; 

e devaluing human skills; and 

e eroding human self-determination. 


All five challenges are relevant for the use of AI in defence and security, but 
some — enabling human wrongdoing, reducing human control, and remov- 
ing human responsibilities — are key to the case in point. 


e ‘Enabling human wrongdoing’ refers to AI systems that foster undue 
biases in the decision-making processes that may lead to erroneous or 
unfair decisions. 

e ‘Reducing human control’ is a pressing challenge given the lack of pre- 
dictability of the outcomes of AI systems. This challenge becomes even 
more pressing when considering the lack of transparency (or opaqueness) 
and explainability of the processes of these systems. Opaqueness and lack 
of explainability hinder human control insofar as they make it hard to 
scrutinise how a given output has been produced, identifying and cor- 
recting errors, as well as auditing AI systems for unethical and unwanted 
consequences. 

e ‘Removing human responsibility’ is problematic as transparency issues 
and distributed design and development processes make it difficult to 
identify the source of errors and unintended consequences of AI systems. 
In turn, this leads to a responsibility gap. 

e Two more challenges have been identified in the relevant literature? 
with respect to the deployment of AI for defence and security purposes. 
These are: ‘escalation’ of activities and ‘lack of control’ (the red circles in 
Figure 7.2). 

e Escalation. The risk of escalation emerges especially in context where 
AI is deployed to support cyber operations. In these cases, AI can re- 
fine strategies and launch more aggressive counter operations. This may 
snowball into an intensification of attacks and responses, which, in turn, 
may threaten key infrastructures of our societies.” 


4 
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e Lack of control. It is worth stressing that the risk related to the lack 
of control should not be confused with any sci-fi scenarios, where ma- 
chines escape human control and overtake humans. These unrealistic 
scenarios divert attention from more concrete and pressing problems, 
like maintaining meaningful control of AI systems, of the cascade effects 
that their use may have, and ascribing responsibilities when deploying 
pervasive, distributed systems, with multiple interactions, and opaque, 
fast-pace execution. 


At a high LoA, ethical problems and related solutions (desiderata) of AI in de- 
fence and security may be mapped against these challenges. However, when 
considering the ethical challenges of AI at the LoApurpose and LoAgthics, it 
becomes clear that the solutions to address them require a more granular 
approach to be effective. Figure 7.2 shows the ethical desiderata for each of 
purpose of use of AI in the defence and security domain defined in the pre- 
vious section. 

The three purposes of use of AI in the defence and security domain are 
more ethically problematic as one moves from sustainment and support uses 
to adversarial and kinetic uses. This is because alongside the ethical prob- 
lems related to the use of AI (e.g. transparency and fairness) one also needs 
to consider the ethical problems related to adversarial, whether non-kinetic 
or kinetic, uses of this technology and its disruptive and destructive impact. 


Uses of Al for National Security and Defence — Ethical Desiderata 


Sustainment and support uses 
e Transparency and fairness 
e Moral responsibility & accountability 
e Human autonomy 
e Protection of rights 
e Robustness Adversarial and non-kinetic uses 
e  Proportionality of outcomes 
e Distinction of targets 
e Meaningful control 
e Redressing 


e Avoid escalation Adversarial and kinetic uses 


e Consistency with Just War Theory 
e Consistency with military virtue 

e Respect of human dignity 

e Foster stability post bellum 


Figure 7.2 A map of the ethical desiderata linked to the specific purpose of the use 
of AI. 


Source: Author’s creation. 
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As shown in Figure 7.2, each category of use has its own specific ethical 
desiderata, but also inherits the ones from the categories on its left. For ex- 
ample, adversarial and non-kinetic uses of AI need to ensure the protection 
of rights, oversight and redressing while limiting the risks of escalation. To 
be ethically sound, these uses of AI need also to respect transparency and 
autonomy, which appear in the sustainment and support category. Similarly, 
adversarial and kinetic uses will have to ensure transparency and autonomy, 
alongside the protection of rights and proportionality, while also respecting 
the principles of Just War Theory, military virtue, human dignity and foster 
stability. Let us now consider in more details some of the key ethical chal- 
lenges of each purpose of use. 


Sustainment and support uses of AI 


Defence organisations already employ Al systems for different non-aggressive 
aspects of operations.” Uses vary from applications in cybersecurity, where 
AI plays an ever-growing role to ensure systems robustness and resilience, to 
Al-based drones capturing video reconnaissance, radio-frequency identifica- 
tion (RFID) tags on food supply.” 

For nations with adequate capabilities, AI systems are likely to reach full 
integration into national defence and security capabilities to support back- 
office, logistics and security tasks. For example, research estimates that the 
number of intelligent sensors in a military setting could reach one million 
per square kilometre similar to the supported connection density of the 5G 
network.” This has been described as the internet of battle things.” In these 
cases, AI will be used to ensure the robustness and resilience of the networks 
as well as to elaborate data and extract relevant information (epistemic tasks). 
All these uses pose serious ethical risks. 

First, consider the use of AI to enhance system robustness. This refers to 
AI for software testing, which is a new area of research and development. It is 
defined as an “emerging field aimed at the development of AI systems to test 
software, methods to test AI systems, and ultimately designing software that 
is capable of self-testing and self-healing.”* 

AI can help with verification and validation of software, liberating human 
experts from tedious jobs, and offering a faster and more accurate testing of 
a given system.*! In this sense, AI can take software testing to a new level, 
making systems more robust. However, we should be careful as societies 
about the way we use AI in this context, for delegating testing to AI could 
lead to a complete deskilling of defence personnel deployed for verification 
and validation of systems and networks and subsequent lack of control of this 
technology. 

Next, let us focus on system resilience. AI is increasingly deployed for 
threat and anomaly detection (TAD). TAD can make use of existing security 
data to train for pattern recognition. For example, in April 2017 software 
firm DarkTrace launched Antigena, which uses machine learning to spot 
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abnormal behaviour on an IT network, shut off communications to that part 
of the system, and issue an alert. These services analyse malware and viruses, 
and some are able to quarantine threats and portions of the system for further 
investigation. 

In certain cases, threat scanners have access to files, emails, mobile and 
endpoint devices, or even traffic data on a network. Monitoring extends 
to users as well. AI can be used to authenticate users by monitoring behav- 
iour and generating biometric profiles, like for example, the unique way in 
which a user moves her mouse around."? Sometimes, this may imply tracking 
“sensor data and human-device interaction from your app/website. Every 
touch event, device motion, or mouse gesture is collected.”*? The risk is clear 
here. AI can improve system resilience to attacks but this requires extensive 
monitoring of the system and comprehensive data collection to train the AI. 
This poses users’ privacy under a sharp devaluative pressure, exposing users 
to extra risks should data confidentiality be breached, and creating a mass- 
surveillance effect.** The sustainment and support uses of AI in defence and 
security pose ethical challenges similar to those related to uses of in AI in 
other domains, like for example the risks to breach privacy. This does not 
make these challenges less important. Indeed, a state actor breaching privacy 
poses severe risks to human right and fundamental democratic values. These 
problems need to be addressed with respect to their merit in the defence and 
security domain, which may complicate their solutions, for they involve bal- 
ancing state interest, national security and respect of individual rights. 

AI is also deployed to enhance situational awareness. Timely, situational 
awareness is decisive to enhance preparedness and to pre-empt threats. How- 
ever, raising such an awareness using AI can be ethically challenging, espe- 
cially given the hybrid nature of threats and different variables at play. This 
is because the threats, which may be hybrid in nature, may also coincide 
with changing facets in the political, economic, strategic, cultural, and social 
circumstances operating around the defender, and attacks can be initiated by 
actors working with changing allies, interests, resources and methods. This 
requires ‘always-on, real-time analytics and anomaly detection capabilities. 
AI offers much to this end, as it enables the analysis of great volumes of data. 
The key challenge is to ensure that large-scale data collection and analyses are 
kept in balance with key regulations and ethical values, to avoid undermining 
civilian trust in defence and security institutions, for example through exces- 
sive, undue surveillance or discriminatory systems. 

AI can extract information to support logistics and decision-making, but 
also for foresight analyses, internal governance and policy. These are perhaps 
some of the uses of AI with the greater potential to improve defence and 
security operations, as they will facilitate timely and effective management 
of both human and physical resources, improve risk assessment, and support 
decision-making processes. For example, a recent chapter by KPMG stresses 
that a defence agency could have only a few minutes to decide whether a 
missile launch represents a threat, share the findings with allies, and decide 
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how to respond. AI would be of great help in this scenario, for it could inte- 
grate real-time data from satellites and sensors and elaborate key information 
that may contribute to the decision-making process. However, the challenge 
is that these uses of AI must ensure that AI systems would not perpetrate 
a biased decision and unduly discriminate, whilst also offering a means to 
maintain accountability and control and foster transparency. 


Adversarial and non-kinetic uses of AI 


The 2019 Global Risks Report of the World Economic Forum ranks 
cyber-attacks among the top five most likely sources of severe, global-scale 
risk.55 The chapter is in line with other analyses about the escalation in fre- 
quency and impact of cyber-attacks,*° and a Microsoft study shows that 60% 
of the attacks in 2018 lasted less than an hour and relied on new forms of 
malware." 

As the threats escalate, so does the need for defence strategies required to 
meet them. The UK and the US have employed ‘active’ cyber defence strate- 
gies that enable computer experts to neutralise or distract viruses with decoy 
targets, and to break back into a hacker’s system to delete data or to destroy it 
completely. In 2016, the UK announced a £1.9 bn investment and a five-year 
plan to combat cyber threats. In February 2020, the UK also established the 
National Cyber Force, as a joint initiative between the Ministry of Defence 
and GCHQ, which is tasked to target hostile foreign actors. On an inter- 
national scale, NATO can now rely on sovereign cyber effects in response 
to cyber-attacks, as agreed at the Brussels Summit.’ This may enable the 
alliance to punish (attributed) attacks and deter attackers from striking again 
in the future. 

AI will revolutionise these activities. Attacks and responses will become 
faster, more precise, and more disruptive. AI will expand the targeting ability 
of attackers, enabling them to use more complex and richer data. Enhancing 
current methods of attack is an obvious extension of existing technology; 
however, using Al within malware can change the nature and delivery of an 
attack. Autonomous and semi-autonomous cybersecurity systems endowed 
with a “playbook” of pre-determined responses to an activity, constraining 
the agent to known actions are already available on the market.*? Autono- 
mous systems able to learn adversarial behaviour and generate decoys and 
honeypots, thus actively luring threat actors,*” are also being commercial- 
ised. Additionally, Al-enabled cyber weapons have already been prototyped 
including autonomous malware, corrupting medical imagery, and attacking 
autonomous vehicles.*! For example, IBM created a prototype autonomous 
malware, DeepLocker, that uses a neural network to select its targets and 
disguise itself until it reaches its destination.” 

As states use increasingly aggressive Al-driven strategies, opponents 
will respond ever more fiercely.” This may expand into an intensification 
of cyber-attacks and responses, which, in turn, may pose serious risks of 


Ethical principles for artificial intelligence in defence domain 169 


escalation and lead to kinetic consequences." To avoid the escalation, it is vi- 
tal that uses of AI respect key principles of Just War Theory which underpins 
international regulations, such as the United Nations Charter, 45 The Hague 
and Geneva Conventions, "° and International Humanitarian Law,” and sets 
the parameters for both ethical and political debates on waging conflicts. 
It will be crucial that the deployment of AI for aggressive and non-kinetic 
purposes respects the principles of proportionality of responses, discriminates 
between legitimate and illegitimate targets, ensures some form of redressing 
when mistakes are made,** and maintains responsibility and control within 
the chain of command. Ultimately, ethical considerations on the adversarial 
and non-kinetic use of AI should contribute to understand how to apply Just 
War Theory in cyberspace and used to shape the debate on the regulation of 
in cyberspace. 


Adversarial and kinetic uses of AI 


When considering the ethical implications of aggressive and kinetic uses of 
AI for defence and security, the main focus of analysis has been the combina- 
tion of AI with machinery that can cause lethal harm to humans and destruc- 
tion to physical objects in a completely autonomous way. 

However, the use of AI for aggressive and kinetic purposes varies, ranging 
from automating various functions of a weapon system to systems that follow 
the pre-programmed instructions of a human, to full autonomy, when the 
weapons system will identify, select, and engage targets without any hu- 
man input. Consider for example, a system developed for the Royal Navy 
called STARTLES which supports human decision making with situational 
awareness software that monitors and assesses potential threats using a com- 
bination of AI techniques. Similarly, the Advanced Targeting & Lethality 
Automated System (ATLAS developed for the US Army support humans 
in identifying threats and prioritising potential targets. Ethical problems vary 
with the degree of autonomy and the ways in which AI might be involved 
in weapons systems, these go from ensuring that AI used to support human 
decision-making for the application of force works correctly to the level of 
autonomy of AWS and control exerted over them. 

The UK government does not possess fully autonomous LAWS and has 
stated that it has no intention to develop them.*” Political actors and mili- 
tary practitioners from other countries have expressed similar commitments. 
Nonetheless, it is important to consider and address the ethical challenges 
posed by fully autonomous LAWS to establish boundaries for the develop- 
ment and use of weapons which incorporate AI but are not fully autonomous 
in their operation. 

A key challenge is to ensure that adversarial and kinetic uses of AI will 
be able to respect the tenets of Just War Theory, for example necessity, pro- 
portionality, and discrimination. So, for example, AI systems must be able 
to distinguish between a member of Armed Forces and a civilian carrying a 
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weapon or recognising the generally-accepted signs of surrender that operate 
in armed conflict. This may be problematic, because AI, at least in its current 
state of development, is insufficiently able to analyse context, in some situa- 
tions its capacity to recognise who is and who is not a legitimate target could 
be significantly worse than that of humans.” 

The responsibility gap is another key ethical challenge. As mentioned in 
Section “Ethical Challenges of AI for Defence and Security Purposes”, whilst 
a responsibility gap is problematic in all the three categories of use of AI, it 
is particularly worrying when considering the adversarial and kinetic case, 
given the high stakes involved.5* This gap becomes an even more pressing 
issue when coupled with the respect of an opponent and of her dignity. Treat- 
ing opponents with respect in warfare is an important way of maintaining 
warfare's morality,55 the interpersonal relation with the opponent is consid- 
ered to be key to this end. Insofar as the use of autonomous LAWS would 
sever this relation, the question emerges as to whether the use of these systems 
undermines the dignity of those whom they target (and possibly also those 
who use them) and lead to a form of morally problematic killing.*° 

Finally, questions arise with respect to the impact of LAWS on interna- 
tional stability. On the one side, LAWS may reduce the time span of the 
hostilities in which states may engage and thus contribute to foster stability. 
They could also be an effective deterrent against possible opponents. On the 
other side, LAWS may lead to unjust war and hamper international instabil- 
ity. This is because the use of LAWS may lower the barriers to warfare?’ pos- 
sibly increasing the number of wars. For instance, it may be the case that the 
widespread use of LAWS would allow decision-makers to wage wars without 
the need to overcome the potential objections of military personnel.°* In the 
same vein, asymmetric warfare that would result from one side using LAWS 
may lead to the weaker side resorting to insurgency and terrorist tactics more 
often.5” Because terrorism is generally considered to be a form of unjust war- 
fare (or, worse, an act of indiscriminate murder), deploying LAWS may lead 
to a greater incidence of immoral violence. 


Ethical guidelines for the use of AI 


As argued in the previous section, the use of the AI in this domain poses 
serious ethical challenges which, if left unaddressed, may lead to disastrous 
consequences for national defence and security and for international stability. 
In this section, we offer five ethical principles, which build on the founda- 
tional bioethical principles identified by Floridi and Cowls°° but are also 
specifically designed to address the ethical challenges linked to the deploy- 
ment of AI in the defence and security domain. The principles specified in 
this chapter refer to both sustainment and support uses and adversarial and 
non-kinetic uses of AI. They should be regarded as the first building block of 
a more comprehensive ethical framework addressing also the adversarial and 
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kinetic uses of AI, which will be the focus of the second, forthcoming, part 
of this project. 

In order to be ethically sound, sustainment and support and adversarial 
and non-kinetic uses of AI for national defence and security purposes should 
respect the following ethical principles: 


I Justified and overridable uses 
II Just and transparent systems and processes 
III Human moral responsibility 
IV Meaningful human control 
V Reliable AI systems 


Justified and overridable uses 


The (non) adoption of AI needs to be justified to ensure that AI solutions 
are not being underused, thus creating opportunity costs; or overused and 
misused, thus creating risks. Similarly, the decision to (or not to) resort to AI 
should always be overridable, should it become clear that it leads to unwanted 
consequences. 

Even when designed and deployed according to ethical principles, AI re- 
mains an ethically challenging technology. Its use may lead to great advan- 
tages for national defence and security. Yet, AI is not a silver bullet. This is 
a lesson that should be learned from the ethical governance of AI for social 
good. As Floridi and colleagues stress: 


it is important to acknowledge at the outset that there are myriad cir- 
cumstances in which AI will not be the most effective way to address a 
particular social problem. This could be due to the existence of alterna- 
tive approaches that are more efficacious or because of the unacceptable 
risks that the deployment of AI would introduce.°! 


At the same time, AI can also encroach upon human rights, International 
Humanitarian Law or pose risks to international stability (the reader will 
recall the risks of snowball effect linked to the adversarial and non-kinetic 
use of Al). This is why the decision to (or not to) delegate tasks to AI systems 
should follow a careful analysis of the risks and benefits in any given context 
of deployment to justify it. 

This principle yields different recommendations when considering sus- 
tainment and support and adversarial and non-kinetic uses. In the first case, 
the principle calls for an assessment of the ethical risks against the expected 
benefits following from the deployment of AI systems. For example, weight- 
ing the benefits of using an AI system that may speed-up a decision-making 
process or optimise logistic and distribution of resources against the likeli- 
hood that it may have a negative impact on jobs and human expertise; or 
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considering the impact on human autonomy when AI is integrated in human 
teams (human-machine teaming). 

When deciding on deploying AI for adversarial and non-kinetic purposes, 
for example for offensive cyber operations, it is essential to ensure that AI 
systems will respect the principles of necessity, humanity, distinction, and 
proportionality.°” This may prove to be a complex task, as the principles of 
International Humanitarian Law are geared towards kinetic forms of war 
waging and therefore their implementation to the case of non-kinetic warfare 
may be problematic. Consider for example, proportionality and the problems 
of assessing the expected damage to intangible entities (e.g. data or services) 
against the concrete military aim to be achieved.°? Satisfying this principle 
will require extending the scope of the fundamental tenets of Just War The- 
ory from kinetic to non-kinetic war waging. A complex but necessary, and 
not impossible, task. 

Given the learning capability of AI and the lack of predictability of its out- 
come, even when uses of AI are justified, a constant monitoring of the ethical 
soundness of the solutions that they provide should be in place. Similarly, 
procedures to override the decision to resort to AI in a timely and effective 
way should be established every time an AI system is deployed. 


Just and transparent systems and processes 


AL systems should not perpetrate any undue discrimination, nor should they 
lead to any breach of the principles of Just War Theory. This is why defence 
and security institutions should ensure that the deployed AI systems, and 
the processes in which they are embedded, remain transparent (and explica- 
ble) to facilitate the identification of the origin of unintended and mistaken 
outcomes, the attribution of responsibilities, and guarantee the possibility 
of scrutinising and challenging processes and outcomes to ensure that they 
remain ethically sound. 
Three aspects are vital to this end: 


e establish processes for ethical auditing; 

e ensure that procured AI systems respect ethical principles; 

e maintain traceability for the design, development or procurement, and 
deployment of AI systems. 


Ethical auditing should involve the entire decision-making process, and so it 
should focus on both human agents and the technological systems, to ensure 
that both agents respect the relevant ethical principles. Transparency of Al 
systems and processes enables access to the relevant information. The former 
requires explainability, while the latter traceability. 

Transparency of AI follows from the effort of designing and developing 
explainable technologies. Thus, it is crucial that in-house and procured AI 
systems are designed and developed with explainability in mind. Defence 
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and security agencies should consider participating actively in the ‘design- 
develop-deploy’ cycle of the AI technologies that they procure and contribute 
to the development phase by setting standards and offering a trusted space 
where these technologies could be beta-tested. To facilitate this process, pro- 
curement policies should account for an ethical scrutiny of the third parties 
involved. 

AI systems are often designed and developed in a distributed way, mod- 
els, data, training and implementation may be managed by different actors. 
At the same time, AI learns by experience: past deployments impact future 
outcomes. This is why transparency requires traceability of sourcing and 
practices, to ensure that the chain of events leading to possible unwanted 
outcomes is not lost in the distributed and dynamic nature of design, devel- 
opment and deployment of Al. 


Human moral responsibility 


Humans remain the only agents morally responsible for the outcomes of Al 
systems deployed for defence and security purposes. While AI systems can 
be considered moral agents, insofar as they perform actions that have a moral 
value," they cannot be held morally responsible for those actions. This is be- 
cause they lack intentionality and understanding of the reward/punishment 
that may result as a consequence of actions. Unplugging an AI system because 
it violated the principle of proportionality, for example, is not a punishment 
for the system. 

However, ascribing responsibilities to humans for the actions of AI systems 
has proved to be problematic, due to the distributed and interconnected ways 
in which AI is developed and the lack of transparency and predictability of its 
outcomes. Two approaches can be followed to enable fair processes to ascribe 
responsibilities: 


1 following the chain of command, control and communication; 
2 faultless, back-propagation approach. 


They can be described more simply as a ‘linear’ and a ‘radial’ approach, re- 
spectively. These two approaches are complementary and serve the twin pur- 
poses of addressing unwanted consequences, mis- and overuses of AI and 
to foster a self-improving dynamic in the network of agents involved in the 
design, development, and deployment of AI for defence and security. 
According to the linear approach, responsibility is attributed follow- 
ing the chain of command, control and communication. In this case, 
decision-makers are held responsible for the unwanted consequences of Al, 
whether these results from failures of AI systems, unpredictability of out- 
comes or bad decisions. In order to ascribe responsibility fairly, it is essential 
that the decision-makers have adequate information and understanding of the 
way the specific AI system works in the given context, of its robustness, of 
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the risks that it may deliver unpredicted (and unwanted) outcomes, of the 
required level of meaningful control, and of the dangers that may follow 
if the AI systems fails to behave according to expectations. The linear ap- 
proach entails a certain epistemic threshold. This means that for the use of 
AI must be coupled with proper training of the personnel both those who 
decide to deploy and those who use it so that they understand the ways in 
which AI systems work, risks and benefits linked to the systems, and the 
ethical and legal implications of the decision to deploy AI. This approach 
rests on the idea that informed decision-makers choosing to use AI do so 
while being aware of the risks that this may imply and take responsibility 
for it. This awareness, in a military context, can help to fill the so-called 
‘responsibility gap’ of Al. 

The radial approach is useful to address unwanted outcomes of AI systems 
that do not stem from bad intentions or follow from actions that are morally 
neutral per se. This approach addresses unethical consequences that spur from 
the convergence of different, independent, morally neutral factors. In the 
relevant literature this has been defined as faultless responsibility.°° It refers to 
contexts in which, while it is possible to identify the causal chain of agents 
and actions that led to a morally good/bad outcome, it is not possible to 
attribute intent to perform morally good/bad actions to any of those agents 
individually and, therefore, all the agents are held morally responsible for that 
outcome insofar as they are part of the network which determined it. 

This is not an entirely new approach, as it is akin to the legal concept of 
strict liability. According to strict liability, legal responsibility for unwanted 
outcomes is attributed to one or more agents for the damage caused by their 
actions or omissions, irrespective of the intentionality of the action and feasi- 
bility of control. When considering human-machine teaming — the integra- 
tion of Al systems in defence and security infrastructures, decision-making 
processes, and operations — what one needs to show to attribute moral re- 
sponsibility according to the radial approach is that 


some evil has occurred in the system, and that the actions in question 
caused such evil, but it is not necessary to show exactly whether the 
agents/sources of such actions were careless, or whether they did not 
intend to cause them. 


All the agents of the network are then held maximally responsible for the 
outcome of the network. The radial approach does not aim at distributing 
reward and punishment for the actions of a system, rather it aims at estab- 
lishing a feedback mechanism that incentivises all the agents in the network 
to improve its outcomes — if all the agents are morally responsible, they may 
become more cautious and careful and this may reduce the risk of unwanted 
outcomes. This becomes quite effective when, for example, the moral re- 
sponsibility is linked to the reputation of the agents. 
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Meaningful human control 


It follows from the previous principle that the deployment of AI should also 
envisage meaningful forms of human control. These will be essential to limit 
the risks that the outcome of AI systems will not meet the original intent, 
to identify promptly mistakes and unintended consequences, as well as to 
ensure timely intervention on, or deactivation of, the systems, should this be 
necessary. 

The concept of meaningful control has been discussed widely in the rele- 
vant literature on LAWS and indeed when considering these systems, control 
is a key element to consider. However, meaningful control is necessary also 
when considering uses of AI that may not lead to the use of force. This is 
because 


military systems must be able to function safely and effectively under 
a wide range of highly dynamic environments and use cases that are 
hard to predict or anticipate during the design phase. They must also be 
resilient to failure and to complex, uncertain and unpredictable events 
and situations where the dynamics of the military domain necessitate 
complex judgements regarding acceptable actions based on rules of en- 
gagement, international law and judgements over legality, proportional- 
ity and risk. Because of this the maintenance of Human Control through 
a combination of specification, design, training, operating procedures, 
and assurance processes is seen as critical in many, if not all military 
systems.°” 


Meaningful human control of AI is characterised as dynamic, multidimen- 
sional and situation dependent and it can be exercised focusing on different 
aspects of the human-machine team. For example, the Stockholm Interna- 
tional Peace Research Institute and the International Committee of the Red 
Cross identify three main aspects of human control of weapon systems: the 
weapon system’s parameters of use, the environment, and human-machine 
interaction.°* More aspects can also be considered. For example, Boardman 
and Butcher suggest that control should not just be meaningful but “appropri- 
ate, insofar as it should be exercised in such a way to ensure that the human 
involvement in the decision-making process remains significant without im- 
pairing system performance.” 

While meaningful control can be dynamic, multidimensional and situ- 
ationally dependent; the principle that prescribes it is only effective insofar 
as it defines a lower threshold below which control is so minimal to be- 
come irrelevant. Hence, the principle can be implemented minimally and 
maximally. Minimally, the implementation of this principle requires hav- 
ing a human on the loop able to understand the functioning of the system 
and its implications and with the ability to ‘unplug’ the system timely and 
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effectively. Maximally, the principle requires individuals in charge of Al 
systems to combine technical, legal and ethical training to ensure that the 
decision to let the system work is informed by all relevant dimensions and not 
a mere vetting of the system. 

Therefore, the principle does not admit fire and forget uses of AI, as it con- 
siders control as an element which can be modulated with respect to a rigor- 
ous risk assessment of unintended consequences, and related negative impact 
on national defence and international stability. Where even lower levels of 
meaningful control cannot be complemented with these assessments, the use 
of AI systems is ethically unwarranted. It should be noted that the principle 
is best implemented when protocols for the attribution of responsibilities for 
misuses of AI and mistakes made by AI systems are in place alongside effec- 
tive redressing and remedy processes. The attribution of responsibility hinges 
on the respect of transparency. 


Reliable AI systems 


The principle mandates the establishing of meaningful monitoring of the 
execution of the tasks delegated to AI. The monitoring should be adequate 
to the learning nature of the systems, and their lack of transparency, while 
remaining feasible in terms of resources, especially time, and hence compu- 
tational feasibility. 

AI has a poor shock response (robustness) and any slight alterations to in- 
puts can degrade a model disproportionately.’” Thus, deploying on AI for 
defence and security purposes could favour opponents’ if the system is not 
deployed according to procedure that envisage forms of monitoring and 
prompt redressing in case of mistakes. This is why this principle prescribes 
the deployment of reliable AI systems, that is systems which are being moni- 
tored throughout their deployment. 

Forms of control may span from new forms of procurement that envisage 
an active role of the defence and security institutions in the design and devel- 
opment process; in house design and development of models; use of data for 
system training and testing collected, curated and validated by the systems 
providers directly and maintained securely; mandatory forms of adversarial 
training with appropriate levels of refinement of models to test their robust- 
ness; sparring training of Al models; monitoring the output of AI systems 
deployed in the wild with some form of in silico baseline model, as suggested 
by Taddeo, McCutcheon, and Floridi.” 

As stressed in the methodology section of this chapter, AI systems are au- 
tonomous, self-learning agents interacting with the environment. Their be- 
haviour depends as much on the inputs they are fed and interactions with 
other agents once deployed as it does on their design and training. Respon- 
sible uses of AI for defence and security purposes need to take into account 
the autonomous, dynamic, and self-learning nature of AI systems, and start 
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envisaging forms of monitoring that span from the design to the deployment 
stages. 


Conclusion 


These principles should not be followed as an algorithm, they do not offer a 
set of instructions that ensure ethically sound decisions with respect to the 
deployment of AI, rather they offer guidelines to spur and articulate ethical 
considerations with respect to the uses of AI in defence and security. For them 
to be effective, it is crucial that the principles are officially adopted by defence 
and security organisations and that a committee or supervising body in charge 
of fostering the adoption of these principles is established. At the same time, it 
is key to train members of staff with respect to the ethical implications of AI. 

Different trade-offs among these principles will have to be defined de- 
pending on the context of deployment. Clearly the trade-offs will have to be 
coherent with the aim to minimise unethical consequences, with the ethical 
values imbued in the practices of defence and security institutions, with re- 
quirements set by laws and regulations, and ultimately with the values and 
rights underpinning our democracies. This is only possible (i.e. the trade-offs 
will be correct) insofar as the humans making the decision are able to take 
into account the principles offered in this document, along with knowledge 
of legal and technical aspects of AI with the goal to reconcile different values, 
interests, and goals. 
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8 Is Stuxnet the next 
Skynet? Autonomous 
cyber capabilities as lethal 
autonomous weapons systems 


Louis Perez 


In 2012, in his first public speech as the new leader of North Korea, Kim 
Jong-Un declared: “The days are gone forever when our enemies could black- 
mail us with nuclear bombs.”! This proud assertion of the Korean leaders 
shows that having nuclear weapons is a strong strategic asset in international 
relations. Therefore, if you assume that one of your enemies is developing nu- 
clear weapons, you will probably try to stop it. This is undoubtedly what the 
designers of the cyber worm Stuxnet had in mind in 2010, two years before 
Kim Jong-Un’s speech, when they designed it to disable the centrifuges at the 
Natanz nuclear power plant in Iran.” Stuxnet has been recognized as one of 
the first cyber-attacks that caused physical damage? and therefore character- 
ized as a cyber weapon.“ Due to its independent functioning without human 
intervention and its automatic spread over the world, Stuxnet has also been 
labelled as an autonomous cyber weapon.’ Autonomy is one of this ambigu- 
ous technological advance that inspire hope and fear. The fear of autonomous 
systems is particularly fuelled by science fiction and the rise of machines. This 
is the plot of the famous Terminator franchise. In these movies, the start- 
ing point is Skynet, an artificial intelligence (AI) system that has developed 
self-awareness and over which humans have lost control. Ironically, this AI 
system was designed to automate American nuclear response. Nevertheless, 
the public did not retain the idea of the AI system but rather their physical 
envelope, the autonomous robots known as the Terminator. Wondering if 
autonomous cyber capabilities (ACC), such as Stuxnet, may soon become as 
the pure fictional Skynet system, the answer is no, considering current and 
near-term technologies. However, it draws attention to how autonomy is 
increasingly being used in cyber capabilities and how States intend to manage 
this in light of potential threats, especially in warfare. 

Autonomy and cyber, and their potential uses in warfare, are two areas of 
concerns for States. Thus, they met regularly in two distinct international 
forums to discuss these topics. As early as 1998, cyber, referred to as in- 
formation and communications technologies, is a topic of discussion within 
the UN First Committee.’ In 2004, a Group of Governmental Experts on 
Advancing responsible State behaviour in cyberspace in the context of in- 
ternational security (UN cyber GGE) was established with 15 members.’ 
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However, it failed to reach a consensus and issue a report. Nevertheless, re- 
ports were subsequently delivered after the creation of others GGE in 2010,” 
2013,10 2015.!! After another lack of consensus at the 2017 GGE, a new GGE 
was created in 2019 and released a report in 2021.17 In parallel with the GGE, 
an open-ended working group (OEWG), open to all UN Member States, was 
established. The OEWG released its final report on March 2021.15 
Autonomy is also a topic in a very specific forum, the Meeting of High 
Contracting Parties to the Convention on Prohibitions or Restrictions on 
the Use of Certain Conventional Weapons (CCW)."* In 2013, States parties 
decided to convene an informal Meeting of Experts to discuss the questions 
related to emerging technologies in the area of lethal autonomous weapons 
systems (LAWS). After three years of informal meetings, States decided to es- 
tablish a Group of Governmental Experts (CCW GGE) on LAWS in 2016. 
After three years of discussions, States successfully agreed on 11 guiding prin- 
ciples on LAWS in 2019.15 Principles especially address issues such as the ap- 
plication of international law, notably international humanitarian law (IHL), 
weapon reviews, responsibility and accountability, human control, etc. 
States remain relatively silent on the interaction between autonomy and cy- 
ber and how international law should apply to it. Indeed cyber and LAWS are 
seen as two different subjects that do not cross. This is even more surprising 
considering the proximity between the both. Indeed autonomous and cyber 
systems have been described as both software-based." The United States De- 
partment of Defense (DoD) science board described cyber as “broadly used 
to address all digital automation used by the Department and its industrial 
base. This includes weapons systems and their platforms.”!® Therefore, what 
is said about LAWS could, in part, apply to cyber systems. In fact, this soft- 
ware base is not ignored from the CCW GGE. One of the aforementioned 
guiding principles acknowledges that LAWS are software-based by calling on 
States to consider “cyber-security against hacking or data spoofing.”!? How- 
ever, it seems that code has been personified in relation to LAWS. Although 
software-based, LAWS are considered physical, whereas cyber systems are 
considered immaterial, despite their physical roots.20 The idea of an interplay 
between autonomous weapon systems and cyber systems appears incongruous 
for States. Yet, beyond the Stuxnet case, cyber systems seem to benefit more 
and more from autonomous capabilities, especially thanks to AI algorithms.”! 
Autonomy promises to enhance system capabilities such as speed and ac- 
curacy in a way that exceed any human capabilities in this area.” Hence, 
in cyber warfare, autonomous systems will be a great advantage, especially 
to execute rapidly mundane tasks but also more complex ones in changing 
environments.” There is an increasing need of adaptative cyber systems 
regarding the necessary work required to ensure adequate defence.” Never- 
theless, clarifications are needed to assess what is an autonomous system. To 
a certain extent, it has been asserted that because cyber systems rely on soft- 
ware and conducted on computers, automation is always present.” However 
one may wonder to what extent automated is part of autonomy and if there is 
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difference from simple automated cyber scripts and very complex ones, such 
as Stuxnet, that are often characterized as autonomous.20 It is unclear if 
autonomy only refers to “fully autonomous” cyber systems or if advanced 
automated system could in a sense also be view as autonomous. Fully auton- 
omous systems that could activate and (re)program itself and replace cyber 
human specialist are far from existing yet“ and Stuxnet was clearly not 
one of them.’ One of the reasons is that great part of cyber activities still 
need humans and can’t be automated, although more and more are.” Given 
the increasingly automation of cyber capabilities and the uncertain point 
that distinguish automation from autonomy, authors have suggested broader 
definitions of autonomy. An autonomous system has been described as a 
system that “generates the rules by which it operates in its environment, and 
that no other entity generates the rules by which the system operates”? or 
the ability of a system “to perform some task without requiring real-time in- 
teraction with a human operator.”*! Several techniques are used to enhance 
autonomous features of cyber systems, especially AI algorithms that allow 
the system to be self-adaptive*” and self-learning.” Therefore, offensive and 
defensive ACC are and will be used in conflict.** Several examples such as 
the DARPA Cyber Grand Challenge demonstrate these capabilities.*° ACC 
can be used in several steps of an attack such as reconnaissance, infiltration 
and command and control.*° 

However, the exact scope of what autonomy is and how concerns related 
to LAWS, including international law, apply to ACC are open questions. 
As a result, the issue of ACC seems to elude both forums despite their cur- 
rent existence and foreseeable developments, notably in warfare. Hence, the 
present chapter will explore an innovative way to apply international law to 
these capabilities by characterizing them as LAWS and subsequently applying 
the CCW GGE outcomes to such systems. The scope of the chapter will 
mainly focus on States positions within CCW debates on LAWS and how 
these positions could apply to cyber means. The analysis will be conducted 
from the various diplomatic positions and interpretations of international law 
expressed by States regarding LAWS definition and IHL compliance. The 
purpose of such an approach is to highlight the advantages and adaptations 
needed for the characterization of autonomous cyber means. For instance, 
what the specific environment of cyberspace would change with regards of 
CCW discussions about LAWS. Moreover, this approach will involve apply- 
ing States’ positions on cyber means in a forum other than the UN GGE and 
OEWG. An useful dialogue will necessarily occur between the States’ inter- 
pretations regarding LAWS and cyber. Therefore, the chapter will demon- 
strate how the States’ positions on ACC as LAWS may impact their position 
on every cyber system. 

In the first part, an analysis of States debates related to the LAWS defi- 
nition will be conducted. Examining States position on the definition 
of LAWS, and on each word composing the expression, it will be con- 
cluded that according to some States’ interpretations within the CCW, 
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ACC could be characterized as LAWS. From this conclusion, another part 
will explore how current debates on IHL regarding physical LAWS would 
apply to such systems in cyberspace. This part will especially focus on 
weapon reviews, human control and IHL compliance of cyber autono- 
mous capabilities as LAWS. 


Autonomous cyber capabilities defined as lethal 
autonomous weapon systems 


This section will describe how some ACC may be characterized as LAWS. 
First, an analysis of States’ discussions on the definition of LAWS will be 
conducted. Noting that almost no State has ever referred to ACC as LAWS, 
it will be examined how States interpretations on each word and concept 
related to LAWS might include cyber means and ACC. The section will 
conclude that, based on some interpretations of the core concepts of LAWS, 
ACC could fit a definition of LAWS. 


The state of the debate: have autonomous cyber capabilities ever been 
defined as lethal autonomous weapon systems? 


The interaction between cyber and autonomy is known from the various 
actors who are involved in these subjects. ACC are sparingly referred to as 
LAWS.*” However, several concerns have been raised about the potential 
threats posed by the interplay between cyber and autonomy. In 2021, in the 
Chair’s Summary of the OEWG, it has been pointed out that “pursuit of 
increasing automation and autonomy in ICT operations was put forward as a 
specific concern.”?8 While some States within the OEWG recognize the in- 
creasing importance of autonomy in cyber systems, they do not refer to them 
as potential autonomous weapons. 

In 2012, the United States DoD adopted the Directive 3000.09 addressing 
autonomy 1n weapon systems. This directive provides a definition ofauton- 
omous weapon systems, however it states that this definition does not apply 
to certain systems, including “autonomous or semi-autonomous cyberspace 
systems for cyberspace operations."% Two elements must be specified re- 
garding this directive. First, it is an undeniable recognition that autonomous 
and semi-autonomous cyber systems exist and are used in cyber operations. 
Second, omitting autonomous cyber systems as part of autonomous weapons 
should not be understood as a rejection from the United States of autonomous 
cyber weapons and was essentially justified on practical grounds. It was ar- 
gued that making a precise classification between these types of autonomous 
systems would have taken a long time while the directive was fast needed to 
establish the US position on the issue of autonomous weapons."! 

Therefore within this international forum and this internal policy, some 
States, acknowledge the interplay between cyber and autonomy and the poten- 
tial uses in conflict but never directly as autonomous weapons. Nevertheless, 
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within the CCW GGE the idea of cyber as AWS emerged. Indeed, experts 
underlined that autonomous weapons will probably be first developed in cy- 
berspace.* They also highlighted the lack of discussion about the fact that 
cyber systems could be autonomous weapons and that cyber weapons also 
lacked human control.” Thus, they urged States to consider these points, 
especially if these cyberoperations have kinetic effects." 

Beyond these experts views, there is one case where a State appears to have 
recognized ACC as LAWS. Indeed, in its commentaries on the operational- 
ization of the CCW GGE guiding principles at the national level, Portugal 
stated: 


Indeed, a hostile actor — be it a State actor, a non-State actor or an actor 
by proxy — in the possession of LAWS may use them as an asymmetric 
tool/mean of warfare or of cyberthreat. For example, LAWS could be 
used for espionage actions, intrusion/infiltration activities or in attacks 
against persons, facilities or networks located abroad, in violation of in- 
ternational law and in an undercover fashion making detection and ac- 
countability difficult or even impossible.%5 


In this statement, Portugal recognized that cyber means could be seen as 
LAWS and used to perpetrate attacks against persons and objects. It is the 
unique occurrence found within the GGE debates where a State directly 
discussed LAWS as cyber means in cyberspace with potential kinetics effects 
on persons or objects. It should be noted that Portugal does not justify this 
point but rather admit it as an obviousness, suggesting that ACC are inherent 
to LAWS. When using the expression LAWS, it could indistinctly refers to 
physical or cyber systems. This view seems somewhat extrapolated giving the 
fact that, apart Portugal, no State has ever defined autonomous cyber systems 
as LAWS. However, this raises the question of whether a definition of LAWS 
could include autonomous cyber systems. 


The debate of the States: the definition of lethal autonomous weapon 
systems and the potential inclusion of autonomous cyber capabilities 


Defining cyber capabilities as LAWS is not an easy task given the fact that 
there is no consensus among States on what LAWS are even though numerous 
States submitted their definitions of LAWS.*° One may argue this is unprec- 
edented in the history arm control treaty negotiations where the definition 
of weapons is not the core of the debate and often come at the end process. 
Within the GGE on LAWS, the definition and the necessity of a definition 
is a debate on itself. This section will consider the stake of the debates and 
analyse, from the States perspective, each term comprising LAWS (“lethal”, 
“autonomous” “weapons systems”) in order to assess if cyber capabilities may 
be understood as part of LAWS. 
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Numerous States recognize that a definition of LAWS could be useful,“ 
and declared it was vital to address potential risks that LAWS could pose.*8 
One argument related to the necessity of a definition is that different defi- 
nitions of LAWS may lead to confusion and misunderstanding.”” On the 
other hand, some States have suggested that having a current definition was 
void of meaning, maintaining that LAWS do not exist and therefore can’t be 
defined. It has also been argued that given the fact that the area of LAWS 
is evolutive and involved dual-use technologies it would be too difficult to 
agree on a common and constant definition.5! 

The purpose of the definition and its link to a regulation was also high- 
lighted. A State indicated that the definition debate “turned into a political 
issue suiting the respective policy positions."5? Another State underlined that 
“a legal definition is generally developed for the specific purposes of a legal 
rule and not in the abstract. Often legal definitions determine the scope of 
a legal rule.” Thus, in order to define LAWS, States must determine what 
are the relevant legal rules and to which systems they apply. Concurrently, 
States recalled that the purpose of defining LAWS should be viewed inde- 
pendently from the regulation5* and should not consist in designing what are 
the “good” and “bad” systems but rather to identify the types of systems the 
GGE should address.” 

Therefore, considering the current debate and the progress achieved on 
many related issues since 2014, without a comprehensive working defini- 
tion, several States argued that, so far, a definition was not necessary for the 
continuation of the work.” Consequently, States agreed upon an approach 
focusing on characteristics and concepts related to LAWS rather than a com- 
mon definition.°* This approach would allow for a common understanding of 
the issues, and avoiding the use of the same words with different meaning and 
overall flexibility with regard to the evolutive nature of new technologies.” 

This position is illustrated by the current mandate of the GGE which refers 
to “emerging technologies in the area of Laws" Interestingly, the man- 
date of the previous informal group was only about LAWS. Following the 
characteristics approach, it has been decided to expand the scope of the GGE 
mandate to emerging technologies rather than solely LAWS which remains 
undefined. Therefore within this forum, States should address all emerging 
technologies that contribute to LAWS. But, as South Africa pointed out, 
there is no definition of emerging technologies which remains an open- 
ended concept.°! Considering this concept, several States specified what this 
could refer. Some highlights it was notably AI whereas others indicate au- 
tonomy.°° The United States recalled that, according to its own definition of 
the term, autonomy in weapon systems is not new and has been used for dec- 
ades.°+ Significantly, no State mentioned cyber as an emerging technology in 
the area of LAWS despite the common software based discussed above. One 
may quickly close the debate of ACC as LAWS, at least as subject to the GGE 
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CCW discussions, by considering cyber means as such emerging technologies 
and therefore including them within GGE mandate. 


The analysis of the lethal autonomous weapon systems characteristics 
from a states’ perspective 


This section will analyse the different meanings, according to States, given 
to the terms which form “LAWS.” The purpose is to determine if the ACC 
could be part of the scope of each element and result in a specific definition 
that incorporates ACC in LAWS. 


“Weapon systems” 


Weapons are at the core of the CCW purposes. The CCW was designed 
in order to regulate and prohibit certain weapons in accordance with IHL 
rules.*5 However, this convention does not define the term “weapons”, fo- 
cusing instead on the effects of the use of a weapon rather than weapon as the 
tool itself. The advantage of this approach is that it provides a flexible scope to 
include new forms of weapons, such as “cyber weapons” and “autonomous” 
weapons. 

“Weapon systems” is the term used in the name and the mandate of the 
CCW GGE. In the quest of defining LAWS, or at least their characteristics, 
States tried to understand the exact meaning a “weapon systems.” In a work- 
ing paper on the weapons review mechanisms, Netherlands and Switzerland 
estimated that “from a tactical point of view, almost anything can be used as 
a weapon.”°° They state that a 


‘weapon’ is not limited to weapons in the traditional sense (firearms, 
artillery pieces, etc.) but includes all objects, devices, etc., [...] provided 
that those objects, etc., are intended to cause harm to persons (including 
injury or death) or damage to objects (including destruction, capture and 
neutralization) (.. ji 


Other States support this assertion. Estonia and Finland, highlighted that 
weapons are not just projectiles but also “other capabilities, such as la- 
sers, high power microwave (HPM), nanoparticles, or other mechanisms, 
[that] could potentially be used to cause harm to an adversary.”°° Another 
State indicates that a weapon system “refers to a weapon along with any 
associated technology necessary for the operation of the weapon.”°? These 
statements demonstrate that a projectile, or even a physical object, is not 
necessary to characterize a weapon. This open a space for including cyber 
means as weapons within the CCW discussion. Indeed, it is recognized that 
what is important is the way and the intent which the named weapon will 
be used. According to this approach, as long as cyber capabilities would 
result in harm or damage, it could be qualified as weapons. Nonetheless, 
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with the exception of Portugal, no State has mentioned cyber capabilities 
as LAWS or weapons. 

Considering this question, several experts recognize cyber means could 
be characterized as weapons. In particular, a rule of the Tallinn Manual 2.0 
characterizes cyber weapons as means of warfare.” The commentaries pre- 
cise that cyber weapons are “used to cause injury to, or death of, persons or 
damage to, or destruction of, objects, that is, that result in the consequences 
required for qualification of a cyber operation as an attack” and “include 
any cyber device, materiel, instrument, mechanism, equipment, or software 
used, designed, or intended to be used to conduct a cyber-attack.”7! Despite 
this manual does not reflect States positions, it is consistent with the previous 
positions mentioned: a weapon is the tool used to conduct an attack, which 
causes harm or damage. 

It is interesting to note that in the UN cyber forums, there is no agree- 
ment on whether cyber means could be characterized as weapons as well. 
The matter is not mentioned in the various reports of the cyber GGE and 
of the OEWG. Because the UN cyber GGE process is quite opaque, there 
is few information from this forum. However, in a 2019, France used the 
expression cyber weapons through its statement.’* Conversely, the OEWG 
is more transparent. In this forum many States refer to weaponization of cy- 
ber means’? while some directly used the expression cyber weapons./4 Some 
States, such as the United Kingdom and Cuba, seem to oppose cyber means 
to conventional weapons.” Eventually, Russia clearly refuses to refer to cyber 
means as weapons.’° 

Recognition of ACC as weapons in the framework of the CCW could 
contribute to broader recognition of cyber means as weapons. Indeed, if the 
CCW GGE considers cyber capabilities as LAWS, or at least if it accepts a 
large definition that possibly includes cyber capabilities, this could be a strong 
argument for accepting certain cyber means as weapons in other forums. 
However, States opposed to this idea in UN cyber discussions are unlikely to 
let this happen within the CCW GGE. 

From this discussion one may conclude that certain ACC may be charac- 
terized as weapons due to potential harmful consequences. However, harms 
or damages could not suffice to qualify such systems as LAWS as it seems to 
require a necessarily lethal effect. 


“Lethal” 


The “weapon systems” mentioned in the CCW GGE mandate are intended 
to be lethal. Even though the word “lethal” is part of the expression, ques- 
tions have been raised regarding the necessity of this lethal element to char- 
acterize LAWS. Several States wondered if AWS should inevitably have to be 
lethal.” 

On the one hand, some States understand lethality as basic and crucial 
characteristics of LAWS beside autonomy. For example, Cuba states “The 
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greater the autonomy and lethality, the stricter the framework that regulates 
them should be.”’® Thus, it has been suggested that the discussion limits only 
to lethal AWS.’ One argument raised in favour of this opinion relied on the 
fact that the 


Inclusion of non-lethal autonomous weapons systems, as a part of LAWS, 
would unnecessarily expand the scope of rules too broadly, which could 
end up driving States to evade such rules for the sake of their own national 
security, and accordingly lead to a loss of effectiveness of the rules.®” 


According to these statements, the lethality of a weapon is an underlying con- 
dition of LAWS, particularly to ensure the effectiveness of their regulation. 
A State specified that lethality means the weapon has a “sufficient payload 
and for means to be lethal.”*! 

On the other hand, numerous States have taken the opposite side consid- 
ering that lethality “should not be conceptually regarded as a prerequisite 
characteristic of autonomous weapon systems.”°” They argue that if lethality 
is retained as an element of AWS, it would be “a novel concept in the CCW 
framework” that was not a prerequisite mentioned during negotiations re- 
lated to blinding lasers and non-detectable fragments.*? Various references to 
IHL were also made underlining that “lethality is not a defining feature of 
any weapon system.”® Indeed, one State argued that “lethality per se is not a 
concept in IHL” and that “lethal weapons may be used in compliance with 
THL.”®° Conversely, another State declare that non-lethal weapon may have 
“lethal impacts in certain circumstances but where the lethal effect is not the 
primary purpose of the system.”*° Therefore, a non-lethal weapon may con- 
travene IHL rules.” Eventually, States criticize that a focus on lethally would 
elude damages to certain goods that are also protected by IHL. 

Regarding this debate, Switzerland recommended to have a broad under- 
standing of AWS 


which would also cover means and methods of warfare that do not nec- 
essarily inflict physical death, but the effects of which may be restricted 
to causing, for example: (1) physical injury short of death, (2) physical 
destruction of objects, or (3) non-kinetic effects.°8 


It also recommended to adopt an IHL-centred approach and to focus on 
“attack” that is a concept defined in the Additional Protocol L.5 An interest- 
ing parallel should be drown with the previous discussion on the definition 
of a weapon. Again, for some States, the landmark should be on systems 
that launch an attack, regardless of whether the system is already qualified as 
weapons or necessarily lethal. Interestingly, Switzerland also mentioned non 
kinetic effect which clearly include cyber dimension.” 

Wondering if a cyber weapon may lead to death, the answer is yes accord- 
ing the experts of the Tallinn Manual”! However, the scope is extremely 
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limited if only cyber LAWS are subject to the CCW GGE mandate. This 
will affect very few systems and probably no current one. Therefore, if one 
want to address effectively ACC as AWS, it is preferable to avoid such lethal 
requirement. 

To conclude, the term “lethality” implies a focus on the effects of AWS. 
Whether lethality is part of the final definition of LAWS or not, ACC may 
be included in such definition. The sole difference will be the number of 
systems that fall within the scope of this definition since fewer cyber systems 
can result in lethality and, so far, none are inherently lethal. 


“Autonomy” 


As we mentioned above, autonomy in cyber capabilities has been asserted by 
some experts. Autonomy is an important element of the CCW discussions 
given that it is often considered as the problematic characteristic that make 
weapon systems potentially against IHL. 

Within the CCW, the definition of autonomy has been the subject of much 
discussion and has brought out opposing views. From this debate it will be 
determined whether ACC correspond in part to the conclusion of this debate. 

One of the main conceptions of autonomy within the GGE discussions 
address autonomy as a spectrum’? that varies “from zero to full autonomy.” 
The categories following this conception often strictly distinguish system 
as automated, semi-autonomous or fully-autonomous. Many States consid- 
ers that LAWS are not automated or remotely operated weapon systems? 
Thereby, some of these States views LAWS as only fully autonomous sys- 
tems.” States indicate that fully AWS does not exist yet” and may never 
exist.” States, such as Cuba” and Russia,”” suggested definition of fully au- 
tonomous and semi-autonomous weapons. 0° However, many States high- 
light that there is no clear reference point for when a system becomes fully 
autonomous”! and that a definition of LAWS should provide the point in the 
scale.1% From this, one can infer the intention to differentiate autonomous 
systems from other systems. However, the determination of what constitutes 
an autonomous system and how it contrasts with other systems is far from a 
consensus. 

Due to this unclarity, States criticized the use of the word autonomy as an 
“on/off” phenomenon!% and underlined it was not a “binary technology.” 104 
It has also been noted that the degree of autonomy “can go back and forth — 
from manual, to semi-autonomous, to autonomous — and just because an 
autonomous system has a manual button does not mean it is not to be con- 
sidered.” !°° Hence, there may be no single general level of autonomy across a 
system.'?° A State affirmed that autonomy “is a relative term, with different 
understandings in different disciplines”! and denigrated the use of the ex- 
pression fully AWS, stating that it seems to imply that there is a clear line, 
rather than a continuum.!%5 Another State added that the expression “fully 
autonomous” was imprecise and potentially unhelpful. 1% From this part of 
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the debates, it can be concluded that cyber AWS might have different degree 
of autonomy and should not be considered as fully autonomous systems, al- 
though the definition of the latter may vary. 

Without rejecting the spectrum conception, many States insisted that what 
matters is not autonomy per se or the autonomy of the system, but rather 
autonomy in function or task and the human involvement in the execution 
of the task.!!? Numerous States indicate that it is primarily autonomy in crit- 
ical functions, notably in the targeting cycle, that is of concerns.!!! Critical 
functions were identified as a good way to overcome issues of defining au- 
tonomy.!!? It was stated it was useful to differentiate this kind of AWS from 
others.!15 Some States argue that autonomy in critical function already ex- 
ists in weapon systems. Il? Another important element raised in assessing the 
autonomy in function, or in the system is the capacity of self-learning and 
self-evolution, especially in critical functions.''? Such autonomous function- 
ality in the operation of a system has been designated of particular concern 
with respect to human control, even though other autonomous systems with- 
out this functionality remains concerning.'!® 

Beyond this technical perspective, many States recall that autonomy should 
also be understood as an interaction between human and machine.!"” It has 
been suggested that this approach will allow to go beyond a technological 
prism that is dependent on the evolution of technology and will ensure the 
implementation of legal obligation.''® For this reason, a number of States 
have called for maintaining a human control over critical functions.'!? This 
position is consistent with the above-mentioned experts’ views of autono- 
mous cyber systems that are referring to the human involvement. 

In sum two main elements seems to emerge from this brief summary of the 
CCW debates concerning autonomy. These two elements are the two faces 
of the autonomy coin:1% the technological face address the technical auton- 
omy of systems that can interact by themselves with a changing environment 
and operate in uncertainty, whereas the human interaction face address the 
control humans have over autonomous systems. However, multiple States 
interpretations and definitions may result from these elements and, so far, 
there is not a clear definition, or even boundaries, regarding the concept of 
autonomy within the cow 

Because the debate is on the concept of autonomy, applying this debate to 
“autonomous” cyber capabilities implies the same issues: is there a spectrum 
from automated to fully ACC or is it more relative? What cyber critical func- 
tions can be autonomous? What control human exerts on such cyber systems 
or functions? 

The Stuxnet case illustrates the various possibilities of characterizing a cy- 
ber system as autonomous or not. Many authors consider that Stuxnet had 
a certain degree of autonomy.'”” Indeed, due to the specific circumstances, 
Stuxnet acted on the local network which was not connected to internet and 
therefore without any direct and external human intervention. It executed 
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critical functions (propagate, identify, infect and control the target, self-up- 
date) by itself to achieved its assigned objective. Therefore, the absence of 
human control as well as its ability to interact with its environment lead 
to characterize Stuxnet as an autonomous system. This lack of human con- 
trol is consistent with the previously mentioned example of fully autono- 
mous system definitions provided by some States. Nevertheless, some authors 
underlined that Stuxnet was not autonomous but rather automated.!?° In- 
deed, Stuxnet did not act autonomously insofar it merely followed steps pro- 
grammed into its code to execute human determined objectives. Once inside 
the local system, the worm surely acted without human control, but human 
reconnaissance and human computing expertise were required to design such 
system.'** Therefore, this demonstrates that depending on the interpretation 
of human control, the vision of the autonomy of a system might vary. If Stux- 
net had capabilities as self-learning or self-evolving, these authors might have 
characterized it as autonomous but it appears that the execution of critical 
functions without direct human control was insufficient to see it as autono- 
mous. One may argue that even if Stuxnet was automated, it made mistakes 
and produced unexpected results, such as the propagation of the worm outside 
Iran.!?° These unexpected results demonstrates that such absence of human 
control is worrisome, even if the system is “only” automated. More broadly, 
discussions surrounding the characterization of Stuxnet as automated or au- 
tonomous illustrates the limits of this semantic debate. The novelty, com- 
plexity and variety of tasks executed by Stuxnet have created confusion on 
its nature and its potential autonomous features. However, whether the sys- 
tem is autonomous or automated, it may lead to critical results and unlawful 
actions. Therefore, the way the CCW parties will define autonomy will be 
crucial because it might exclude automated systems that could have unlawful 
consequences. 

In any case, regarding the characterization of cyber capabilities as LAWS, 
whatever the definition of autonomy adopted by the CCW, some cyber sys- 
tems will always seem capable to fit such definition due to the software based 
of such systems. If the autonomy is somewhere in a system or a function, it 
will partly come from the software and therefore linked to cyber characteris- 
tics. Thus, this debate does not preclude to characterize ACC as autonomous 
in the sense that some States understand and interpret autonomy within the 
framework of the CCW. 

From all of the above, the characterization of ACC remains an open pos- 
sibility. Indeed, due to the variety of interpretations for each concepts related 
to LAWS, there is space where cyber capabilities can fit. It is interesting to 
note that for each concept, a bottom-up approach was suggested to avoid 
definition issues. Following this potential characterization of cyber means 
as AWS, the next part of the chapter will analyse what are the implications 
of applying the GGE CCW outcomes to such systems with regard to IHL 
compliance. 
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Cyber autonomous weapon systems and international 
humanitarian law 


Considering ACC as AWS, also referred to hereafter as cyber AWS, implies 
that such capabilities are subject to the CCW GGE mandate and therefore 
affected by the discussions. Many topics are at the forefront of the discussions 
within the forum: mainly IHL, but also responsibility and accountability, 
ethics, international security issues, etc. Thus, the choice has been made to 
focus on IHL in this chapter. This choice relies on several reasons. The inex- 
tricable link between the CCW and IHL encourages to primarily focus on it. 
Indeed, a great part of the debates address IHL issues, such as applicability or 
weapon review. In addition, dealing with IHL issues can help anticipate other 
issues. For instance, ensuring IHL compliance would have consequences on 
responsibility and ethics. Therefore, the next sections will address the ap- 
plicability and the application of IHL to ACC in light of the CCW’s GGE 


discussions. 


The applicability of international humanitarian law to cyber 
autonomous weapon systems 


Not surprisingly, the GGE recognizes the applicability of IHL to LAWS in 
its first principle." Indeed, CCW is based on IHL and its material scope 
relies on Geneva Conventions.'?’ Thereby, admitting certain ACC as LAWS 
will conduct to fully recognize that IHL applies to some cyber means of 
warfare. 

However, the IHL applicability to cyber means of warfare has not always 
been explicit and is still a source of discussions. NATO,!7 European Un- 
ion!?? and numerous States!3 recognized the applicability of IHL to cyber 
operations during armed conflicts. This has also been demonstrated by ex- 
perts of the Tallinn Manual.'*! Notwithstanding, several States refused to 
recognize such applicability. Within the UN cyber GGE the applicability 
of international law and especially IHL to cyberspace is a recurrent matter. 
A major step was taken in the 2015 report of the UN cyber GGE where 
States, “note[d] established international legal principles including, where 
applicable the principles of humanity, necessity, proportionality and distinc- 
tion.” 8? Without naming it, States recognized that the core principles of IHL 
apply in some, unspecified, circumstances to cyberspace.!* 

However, difficulties tarnishing this success emerged in the discussions 
that followed. Despite the discreet reconnaissance of IHL’s applicability in the 
2015 report, several States, like China, Cuba or Russia, objected to use the 
term of IHL. This position mainly relied on the argument that the applica- 
bility of IHL “would legitimize a scenario of war and military actions in the 
context of ICT.”!** This contributed to the failure of the UN cyber GGE to 
adopt a report in 2016—2017. Interestingly, the same type of argument was 
used within the CCW discussions. The report of an informal meeting of 
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experts on LAWS indicates that “several delegations cautioned against taking 
for granted that existing IHL applies to LAWS and that by doing so, these 
weapons could be prematurely legitimized.”!*° Again, several States seem 
to view the applicability of IHL as a potential threat due to legitimization 
such means in wartime. The legal (ir)relevance of such arguments, which 
seems more political than legal,!°° will not be discussed here. Thereafter, the 
OEWG Chair’s report highlighted that States recalled IHL “neither encour- 
ages militarization nor legitimizes resort to conflict in any domain.” 15 

This undeniable applicability of IHL to LAWS, and in this paper to cyber 
AWS, could push States to recognize the applicability to other cyber means 
of warfare. Indeed, the recognition of the applicability of IHL to cyber AWS, 
implies a recognition of the applicability of IHL in cyberspace is those specific 
circumstances. This would demonstrate the inconsistency of the political ar- 
gument mentioned above. In that sense, efforts have been made recently. In 
2021, the report adopted by the UN cyber GGE indicates that “international 
humanitarian law applies only in situations of armed conflict.” "38 While IHL 
is finally mentioned, it is recognized as applicable “only” in armed conflict. 
Even though IHL is the law of armed conflicts, several IHL obligations apply 
in peacetime such as the legal review of new means and methods of warfare. 
Hence, the wording adopted by the GGE may be confusing regarding the 
scope of IHL rules that apply to cyber means. Again, the work of the CCW 
GGE is useful in this regard. States have recognized that the compliance with 
IHL is not limited to the rules related to the conduct of hostilities?’ and 
that peacetime IHL rules should be taken into account.!* The application of 
the CCW GGE discussions regarding the applicability of IHL rules to cyber 
AWS is therefore helpful to support the applicability of all the relevant IHL 
rules to cyberspace in these circumstances. A State which admits this appli- 
cability of IHL to LAWS could not deny it to any cyber means of warfare 
because LAWS would include cyber AWS. 


The application of international humanitarian law to cyber 
autonomous weapon systems 


If the applicability of IHL could seem a less striking issue, the question of how 
IHL should apply is a major one. Whether for cyber operations or LAWS, 
States had discussions about how IHL rules should apply. Regarding cyber, 
the Tallinn Manual brings several answers and interpretations of IHL rules. 
However this document only express experts’ views. States may have differ- 
ent opinions. Thereon, the last GGE requested States, on a voluntary basis to 
share their understanding of how international law applies. The report was 
released in July 2021.14! 

IHL application is also one of the main subjects within the CCW. Main 
concerns are related to the obligation to conduct a weapon review and to 
assess the way LAWS comply or not with IHL principles. These two subjects 
will be analyse from a cyber AWS perspective. 
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The legal review of cyber autonomous weapon systems 


According to the article 36 of AP I, 


in the study, development, acquisition or adoption of a new weapon, 
means or method of warfare, a High Contracting Party is under an ob- 
ligation to determine whether its employment would, in some or all 
circumstances, be prohibited by this Protocol or by any other rule of 
international law applicable to the High Contracting Party. 


In sum, a State has the obligation to assess the compliance of a new weapon to 
international law, especially IHL. This article is not considered as part of cus- 
tomary international law and therefore only applies to the 174 States parties 
to AP 1.'4 However, customary international law still requires from every 
State to ensure the means of warfare used complies with IHL rules, whether 
customary rules or treaty rules.'*4 

In this part, the CCW GGE weapon review mechanism discussions will 
be mainly analysed with regards to its potential contributions to international 
law and cyber discussions. Thus, it will not examine in depth the conditions 
and content of such legal review regarding cyber AWS. Nonetheless, the 
main question of cyber AWS compliance with IHL will be address in the 
next section. 


The weapon review debates and cyber capabilities 


One of the major merits of CCW GGE discussions has been to highlight 
the weapon review mechanism, notably mentioned in the guiding principle 
(c).! This spotlight on the legal review requirement brought a lot of debates 
on what exactly is this review and how to apply it to LAWS. Indeed, char- 
acteristics of LAWS such as autonomy and self-learning capacity result in a 
potential unpredictably which force States to ensure the proper functioning 
of the system through such a weapon review.'*° 

Despite the usefulness of the weapon review in ensuring IHL compliance, 
several States argue that this obligation relies on national procedures that 
will result to a lack of international uniformity and different standards.“ To 
address this problem, many States suggested that there should be a voluntary 
exchange of best practices on the conduct of the legal reviews on emerg- 
ing technologies in the area of LAWS.’ Argentina particularly emphasized 
that this sharing could be based on article 84 of the PA I which encour- 
ages such an exchange of information.!* This exchange was described as a 
confidence-building measure that could enhance transparency and trust in 
the use of LAWS and compliance with IHL.15 It was proposed that this pro- 
cess relied on a specific mechanism!*! where States would share information 
on their national implementation of the weapon review.152 This exchange 
could then lead to a comparative analysis.°? Demonstrations of LAWS have 
even been suggested, !>4 However, States recalled there was no obligation 
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to share information about the review and its results.!°° The limits of na- 
tional security and industrial property rights in sharing information were also 
raised.!°° More pessimistically, Austria indicates information was “unlikely 
to be shared in real time with the broader international community”! and 
other States pointed out it will be “difficult to assess the quality of weapon 
reviews given only a limited number are publicly available 155 Notwith- 
standing, several States already shared information on their weapons reviews 
like Australia, Germany,” Netherlands, !®! Russia, Sweden! or the 
United Kingdom.10% Considering this dynamic and with regards to cyber 
AWS, one may assume that States will be encouraged to share information 
on the specific case of ACC if there are defined as AWS. In addition, these 
information exchanges, referred to as confidence-building measures, can en- 
hance those already discussed in UN cyber forums.!©° 

It’s noteworthy that even if States refuse to define cyber AWS as LAWS, 
the current debates within the CCW remind to States their duty to review 
new weapons, means and methods of warfare. Therefore, the work and de- 
bates on how to assess autonomous systems such as LAWS could prove to be 
useful for States to evaluate autonomous and non-autonomous cyber means 
of warfare. Even though the experts of the Tallinn Manual provide a con- 
structive analysis of the weapons review to cyber means of warfare,!% States 
remained relatively silent on that matter which is hardly discussed within 
UN cyber forums.'©” In 2021, in accordance with the call of the cyber GGE 
to States to share their national views on how international law applies in 
cyberspace, Australia recognized that “[a] cyber capability could, in certain 
circumstances, constitute a ‘weapon, or a means or method of warfare’ within 
the meaning of Article 36 and require a review.” 168 One may suppose this a 
result of the CCW GGE dynamic as Australia demonstrated to be very active 
on article 36 matter.'°? Other States such as Brazil and Switzerland also rec- 
ognized the necessity of article 36 to cyber means." 

Furthermore, recalling the discussion with the expression “emerging tech- 
nologies in the area of LAWS,” one may consider that cyber is one of these 
emerging technologies contributing to LAWS. This is obvious if LAWS in- 
clude cyber AWS, as in this chapter, but it appears less obvious for physical 
AWS as their software base is often omitted. Thus cyber capabilities used in 
AWS, whether cyber or physical, should be assessed in the weapons review. 


Perspectives on the application of the weapon review to cyber 
autonomous weapon systems 


The cyber dimension of cyber AWS will impact the legal review. Such a re- 
view promises to be very difficult because it combines two technologies for 
which there are already many issues.!7! This would require an entire chapter, 
thus the present one will just share some perspectives on that topic here.17? 
Addressing the question of who should conduct such review States within 
the CCW GGE debates required this should be done by a multidisciplinary 
team.” Considering cyber AWS, the weapon review team should include 
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cyber and AI experts beside the legal ones. On the elements that needs to be 
reviewed regarding AWS, specific computer hardware and software, data, 
environment of use have to be assessed. This may be challenging given the 
fact that cyber data and environment are very different from physical ones. 
With respect to the timing the review should take place, cyber AWS reviewer 
will have to deal with the recurrent updates of the cyber AWS software, 
probably more than physical ones. The self-learning and self-adaptative al- 
gorithms that may be used in cyber AWS are also of concerns.!/* For in- 
stance, if a cyber AWS changes its own code, it should probably be reviewed 
again.!”° Therefore, a cyber AWS acting without direct human intervention, 
like Stuxnet, should not have such capabilities because a review would be 
required but impossible.!”° 


Human control over cyber autonomous weapon systems and 
international humanitarian law compliance 


Interestingly, human control has rarely been mentioned with respect to cyber 
systems.'’’ Conversely, it is one of the core topics within the CCW GGE, 
notably addressed by the guiding principle (c). Human control raises the 
question of the extent of human-machine interaction required to comply 
with IHL. There are two opposing positions on this subject.'7? On the one 
hand, following a technological approach, States argue that human control 
relied on technical characteristics. If IHL assessments can be programmed in 
a AWS, there is no problem, especially considering the alleged superiority 
of machines over humans. This position, however, does not exclude human 
from any control but adapt humans involvement regarding technical means. 
On the other hand, following a more humanistic approach, States believe that 
IHL assessments require human judgment in all cases.'®° While technologies 
can help humans comply with IHL, machines can’t enforce IHL rules. This 
debate exceeds the LAWS discussions and applies to technology in general. 
With respect to cyber warfare, it seems that the first view is commonly shared. 
At least it seems rarely argued that a cyber system requires human judgement 
in its conduct.'®! The difference probably lies in the context of utilization. In 
war time, this position may change especially considering ACC. The present 
section will explore the relation between cyber AWS and IHL. 


Cyber AWS and the enhancement of LHL 


Fear of a loss of control of AWS and potential subsequent violations to IHL 
rules usually surround the CCW debates. Against the flow, some States, sup- 
porting the technological approach, point out that AWS could also be a better 
way to ensure IHL. This position is reflected in guiding principle (h) stating 
“consideration should be given to the use of emerging technologies in the 
area of lethal autonomous weapons systems in upholding compliance with 
IHL and other applicable international legal obligations.” !*” 
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In 2018, the United States provided a working paper on “Humanitarian 
benefits of emerging technologies in the area of lethal autonomous weapon 
systems.” !89 Throughout the document the State describes functions and ex- 
amples of ways in which LAWS could reduce risks to civilians such as au- 
tonomous self-destruct, self-deactivation or self-neutralization mechanisms, 
increase of military awareness due to analysis of data, improvement of the 
assessments of the likely effects of military operations, automation of target 
identification, tracking, selection and engagement and the use of force in 
self-defence. In other reports, benefits such as the improvement of precision 
and accuracy of systems | 84 
erations and the protection of civilians have been highlighte 

One may wonder the result of such position regarding cyber AWS. It is 
interesting to note that most of the benefits raised about LAWS do not rely 
on physical space. Indeed a cyber AWS could also have self-destruct, self- 
deactivation or self-neutralization mechanisms, analyse data, conduct likely 
effects assessment or autonomously identify and engage a target. Of course, 
benefits requiring physical means such as mine clearance or civilian rescue 
does not apply, except following an analogy in the cyberspace. Conversely, 
cyber AWS would provide hereabove benefits of autonomy plus those solely 
related to cyberspace. Interestingly, this position emphasizes the fact that 
these technologies are not necessarily bad and dangerous, but can bring ben- 
efits. However, most of the debate focuses on the risk of IHL violations. 


as well as tasks such as mine clearance, rescue op- 
185 
d. 


Cyber AWS, human control and the respect of LHL 


The more systems become autonomous, the more fear of loss of human con- 
trol increases. Human control and human-machine interaction have been key 
concerns within the CCW GGE debates. Even though there is a consensus on 
the necessity to ensure human control regarding predictability and the need 
of human in some decisions for IHL compliance, '*° States recognized there is 
still a need to determine the type and extent of human involvement or con- 
trol.!8” As stated in the guiding principle (c), human-machine interactions are 
found at various stages of the weapon life cycle. States recognized that human 
control is not limited to targeting but cover a larger spectrum from research 
and development to post-use assessment.!5% Nevertheless, in this section, the 
focus will be specifically on the human control relating to the use and the 
targeting process of AWS regarding IHL rules. 

A subsequent debate of the one opposing technological and humanist ap- 
proaches are the nature of human control: direct or distributed. Some States 
argue there should be a direct human control where human is involved in the 
“concrete decisions related to the ‘when and where’ of the use of force need 
to be taken,” !®? the maintenance of communication links with the chain of 
command,'”” or even direct control and supervision of humans at all times 
and all steps.1?! One may understand the reasons of such a position that en- 
sure the constant presence of humans in the operation. However, this may be 
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impracticable, especially for cyber AWS. Cyber means are rarely under the 
direct supervision of humans at all stages of the operation and at the targeting 
moment. For instance, Stuxnet worm acted without direct human control, in 
particular when it hit the nuclear facilities. Even concerning physical AWS, 
direct human control has been criticized. The United Kingdom stated that 
“solely relying on an operator making decisions in the heat of the moment as 
a panacea for human control is never the safest approach.” 1°? For these States, 
human control as a distributed control is more relevant. In this configuration, 
control is shared among different actors (commander, information analyst, 
pilot, etc). Following the idea that direct control is not necessary, some 
argue that more autonomy could be granted to systems if humans are at least 
present at one moment of the Spanii (especially at the programming stage) 
and IHL compliance is assessed.’ 

The type of human control requirement that will be adopted by CCW 
parties is therefore important because AWS will not act in the same man- 
ner depending on the control chosen. The issue of speed illustrates this. In 
some circumstances, States agreed that a system may be used autonomously 
if humans are too slow and the response is urgent. °’ The United States indi- 
cated that several existent defensive systems are already being used with much 
greater speed and accuracy than a human could achieve manually.'”° There- 
fore, time constraints may preclude direct human intervention and decision, 
and impose to rely on AWS.!” This position is particularly welcome with 
respect to cyber AWS. Indeed, cyber operations take place in cyberspace with 
fewer constraints than in the physical world. Hence, some cyber operations 
occur at higher speed than a human can observe.'”® If States reject this posi- 
tion and require a cyber AWS operation at a human speed in order to ensure 
human supervision, no benefit would remain. A question that is left open is 
whether this potential acceptance for speed is only for defensive systems or 
whether it will extend to offensive means. In cyberspace, some argue that 
facing cyber autonomous defensive means, States will be more inclined to use 
such systems in offense. 

Whether offensive or defensive AWS, States suggested, in accordance with 
principle (c), several pre-programmed operational constraints as a means to 
ensure human-control and IHL compliance, especially principle of precau- 
tion.2% The suggested constraints are notably related to the selected tasks, 
controls on the environment (target profiles, time frame, movements in 
an area, operating environment), parameters of use (deactivation, fail-safe 
mechanisms), means of interaction (overriding, abortion of the task, means of 
communication, monitoring and recording information mechanisms, limits 
on self-learning capacities).7”' However, not all States agreed on each of these 
constraints, especially on those requiring supervision and communication 
at all times. Given these constraints, States concluded that there is no “one- 
fit-all” human control requirement and that human-machine interactions 
should be established on each single use of Aws?” Many of these con- 
straints are particularly suitable to cyber operations except those related to 
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physical environment like scope of movements.” For instance, the design 
of cyberoperation often requires to identify target profile. Moreover, cy- 
beroperation are often unique and designed towards a precise goal.204 Such 
unique design corresponds with the necessity of an ad-hoc human-machine 
assessment. Nonetheless, requirements such as the time frame, operating en- 
vironment, overriding capabilities, permanence of means of communications 
may be difficult to design in cyber AWS without reducing their efficiency. 
Thus, if States agree on these requirements, this would reduce the possibility 
of potential lawful cyber AWS. 

From this, it is interesting to note the relation between cyber AWS and 
IHL rules discussions within the CCW GGE. Because the CCW GGE does 
not address directly cyber AWS, few information can be extracted from the 
forum on this subject. So far, most States approach the question of AWS and 
IHL from the perspective of physical space. Therefore, it is the difference of 
nature between physical and cyber space that will induce adaptations. For ex- 
ample, the cyber operational environment may be less chaotic, at least more 
deterministic, than the real world and therefore cyber AWS may act with 
more predictability.” During the target process, the cyber AWS might face 
less difficulties due to the fact it relies on different sensors with less uncertain- 
ties, for example image recognition. ®® Similarly, concerns related to physical 
situations of individuals such as denial of quarter?” or the incapacity to rec- 
ognize the surrender of a combatant?%5 do not apply to cyber AWS. On the 
other hand, the cyber space implies new elements to take into account when 
addressing AWS. If the environment is more deterministic and predictable, 
an error can be more decisive: one weakness might compromise the whole 
system or operation. Also, combatants are naturally out of the cyber scope but 
new objects are of concerns such as every connected infrastructure, especially 
dual use ones, or data.2% In the same vein, an author pointed out the diffi- 
culty to suspend the protection due to civilian who participate to hostility 
by using cyber AWS.*!° Thus, it has been mentioned that adapted methods 
have to be developed for cyber AWS. For example, one author specified that 
to comply with principle of precaution, specific methods of reconnaissance, 
coordination and patching are needed to suit the cyber dimension.”"! Also, 
the fact that Stuxnet had evaded from to other computers are of concern re- 
garding the geographical scope of IHL.7!* 

These few elements justify the need for further discussions on cyber AWS 
and compliance with IHL. To date, they are not discussed by States either 
within the UN cyber forum or the CCW GGE due to the implicit refusal to 
recognize the potential threats to arising from ACC. 

It is a matter of fact that autonomy will be used more and more in cyber 
warfare. The increased development of computing technologies, including 
AI, is addressing the need for faster, stealthier, innovative, and destabilizing 
cyber capabilities. Such methods will change the way cyber conflicts are 
conducted. However, improving cyber capabilities cannot be achieved in an 
unbridled manner. A lack of human control over increasingly ACC cannot 
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be done outside of any regulatory framework, especially the legal one. 
Therefore, the present chapter demonstrated in an innovative way that, in a 
sense, ACC could be characterized as LAWS. The purpose of such demon- 
stration was to find a forum where States could discuss the future of cyber 
conflict and the autonomous dimension of cyber capabilities in warfare. So 
far, the CCW GGE is the only forum that addresses autonomy and conflict. 
The simplest way to provide a regulatory framework for CCA was therefore 
to apply the CCW GGE discussions by characterizing them as LAWS. The 
vast range of States positions on LAWS definition and related concepts allow 
to consider such assertion. From this, the analysis of ACC as cyber LAWS 
results in several consequences for States’ positions regarding cyber means. 
In sum, the recognition of ACC as LAWS stimulates the debates on some 
current cyber issues like weapons characterization, legal review, applicabil- 
ity and application of IHL and human control. Furthermore, this particular 
exercise leads to consider the specific consequences for international law, 
especially IHL, regarding interactions between cyber and LAWS. While 
it is unlikely that Stuxnet will result in a Skynet system that endangers 
humanity, the use of ACC in warfare is a concern for future cyberconflicts 
and should be address by States whether in the UN cyber forum, the CCW 
GGE one or a new international forum. This thought-provoking chapter is 
intended to draw States’ attention to this issue. The subject has been put on 
the table. It will be back. 
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9 Advanced artificial intelligence 
techniques and the principle 
of non-intervention in 
the context of electoral 
interference 


A challenge to the “demanding” 
element of coercion? 


Jack Kenny 


The nature of foreign interference in the affairs of another state in the current 
era is fraught with complexities. Globalisation and advances in technology 
have enabled more opportunities and methods by which to interfere in the 
domestic affairs of foreign states. W hile traditional news media was controlled 
by domestic news organisations through broadcast and print, the availability 
of personal computers and smart mobile devices have revolutionised access to 
information and how news is consumed. The emergence and popularity of 
social media platforms on which users are able to publish their own content 
and share links to news articles has enabled state-sponsored disinformation 
campaigns to exert an unprecedented impact on public discourse in other 
states, creating “echo-chambers” where users are exposed to disinformation. 
The algorithms of online platforms repeat and reinforce those views, exacer- 
bating extreme political positions.' 

Beyond traditional influence operations, deepfakes of prominent public 
figures have prompted much discussion about the role of advanced artifi- 
cial intelligence techniques in disinformation campaigns.” Not only have 
advances in technology and the invention of the internet amplified efforts to 
influence an electorate without the consent of the target state, technology has 
also lowered the technical barrier of capabilities required for conducting such 
operations and effectively rebalanced the traditional power dynamic of cer- 
tain states in their ability to influence others. For example, democratic states 
with less restrictions on access to online platforms who support the freedom of 
expression are more susceptible to electoral interference on online platforms 
than states who have tighter controls or restrictions over online access and 
the sharing of information, and who do not hold free and fair elections. The 
development and availability of open-source artificial intelligence software 
for content creation has removed barriers relating to expertise and hardware 
requirements, enabling anyone with time and access to cloud computing to 
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run sophisticated machine learning processes and graphical rendering.? The 
technology for advanced artificial intelligence techniques is improving at a 
rapid pace, for example, experts predict that deepfake videos may soon be in- 
distinguishable from genuine videos." As technology advances and the means 
to use advanced artificial intelligence techniques become more widely avail- 
able, their use in online disinformation campaigns is expected to become 
increasingly prevalent.’ Given the purported effectiveness of disinformation 
campaigns on online platforms and the difficulties faced by governments 
and social media companies in how best to tackle such content, advanced 
artificial intelligence techniques hold the potential to be incredibly effective 
and influential in spreading disinformation, further enhancing the ability of 
foreign actors to interfere in the domestic affairs of states. 

The principle of non-intervention is an established rule of customary in- 
ternational law that is the corollary of the principle of sovereignty. Closely 
linked with the concept of a state’s domestic affairs, also known as domaine 
réservé, and the international legal limits on a state’s jurisdiction,’ it prohibits 
a state from intervening by coercive means in matters within another state’s 
sovereign powers. 

There is currently widespread agreement that, in principle, existing inter- 
national law applies to state cyber operations.® Reports of the UN Group of 
Governmental Experts on Developments in the Field of Information and Tel- 
ecommunications in the Context of International Security (GGE), adopted 
by consensus, specifically confirm that the principle of non-intervention ap- 
plies to state cyber operations.” Statements by a number of states on the appli- 
cation of international law to cyber operations further support the application 
of the principle of non-intervention to cyber operations,” though statements 
that have been published to date remain general and broad in nature. 

This chapter seeks to examine the application of the principle of non- 
intervention in international law in relation to the use of advanced artificial 
intelligence techniques in disinformation campaigns of electoral interference. 
As artificial intelligence technology advances, techniques such as deepfakes 
possess characteristics that have the potential to be extremely influential in 
interfering in the affairs of another state by means of deception in various 
forms. This chapter uses these techniques as a case study to challenge and 
explore the limits of the coercion element required to establish a prohib- 
ited intervention. In the context of cyber operations, it has been argued that 
the non-intervention principle in international law involves a relatively high 
threshold of violation because of the requirement for the behaviour on the 
part of the perpetrating state to be coercive in nature.'' This chapter makes 
the argument that the non-intervention principle is capable of a broader 
application where coercion requires a “degree of pressure” to deprive the 
target state of control of its state functions. While the use of sophisticated 
technological methods such as advanced artificial intelligence techniques in 
influence operations do not change or alter the criteria on which coercion is 
evaluated, they amplify the scale of effects that are possible to achieve from 
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indirect interference operations which increases the likelihood that such ac- 
tivity could interfere with a democratic state’s inherent right to run free and 
fair elections and may therefore constitute a prohibited intervention. In that 
sense, the chapter challenges the understanding that mere influence alone is 
not capable of being coercive and argues that influence operations are capable 
of constituting a prohibited intervention. 

This chapter consists of three substantive sections. The first section will 
introduce the principle of non-intervention and discuss its status as a rule 
of customary international law, including the identification of elements re- 
quired for a prohibited intervention, consideration of the scope of what is 
encompassed within a state’s domaine réservé, a discussion of common termi- 
nology and a brief overview of state practice that has frequently been at odds 
with the rule. The second section of the chapter will attempt to define the el- 
ement of coercion and its application in the cyber context, including an over- 
view of sources and debates in both non-cyber and cyber specific literature 
and an examination of the position of states, and the relevance of intent and 
outcome in such scenarios. The third section will discuss these arguments in 
relation to electoral interference and the challenges presented by advances in 
artificial intelligence techniques that are increasingly prevalent in enhancing 
the effectiveness of online disinformation campaigns. The chapter will then 
offer conclusions on its content and analysis. 


The principle of non-intervention 


The principle of non-intervention!” is an established rule of international 
law that has been confirmed by the International Court of Justice (ICJ) to 
be ‘part and parcel of customary international law’.!? The principle of non- 
intervention is an inter-state doctrine, and as such does not apply to the ac- 
tivities of non-state actors unless the activities of such actors can be attributed 
to a state.! For the UN and UN member states acting through its organs, the 
prohibition of intervention in essentially domestic matters is set out in Article 
2 (7) of the UN Charter.!5 There is a close relationship between the princi- 
ple of non-intervention and the use of force, though ‘[w]hile the customary 
rules of international law relating to intervention have now to a considerable 
extent to be considered alongside the more general prohibition on the use of 
force, intervention is still a distinct concept.’!° While the principle of non- 
intervention is recognised as part of customary international law,” the prin- 
ciple has been described in the literature as vague and ‘elusive’.® In particular, 
while the prohibition applies to both interventions by force and non-forcible 
interventions, the application of the rule is not clearly defined outside of sce- 
narios involving use of force,” where ‘much depends on context, and even 
on the state of relations between the states concerned’.”” 

In the Nicaragua case the IC] confirmed that the non-intervention principle 
is ‘part and parcel of customary international law’, while also recognising that 
‘examples of trespass against this principle are not infrequent’.”! The Court 
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further elaborated on the legal effect of states’ non-compliance with the prin- 
ciple, explaining that ‘[i]t is not to be expected that in the practice of States 
the application of the rules in question should have been perfect, in the sense 
that States should have refrained, with complete consistency, from the use of 
force or from intervention in each other's internal affairs.” The Court also 
noted that ‘[e]xpressions of an opinio juris regarding the existence of the prin- 
ciple of non-intervention in customary international law are numerous and 
not difficult to find.” Lowe notes the difficulty in reconciling the pervasive 
practice of intervention with the non-intervention rule.” 

While states frequently condemn the acts of other states as intervention in 
their internal affairs,”° there is clearly a ‘long-standing contrast between the 
word and deeds’, where ‘non-intervention is preached, but not practiced’.”° 
In this sense, state practice and opinio juris concerning the non-intervention 
principle are often at odds with each other,” with the frequent breach of the 
principle even leading some to question its status as law.” 


Elements of a prohibited intervention 


In the Nicaragua case, the ICJ identified two elements that are required for an 
unlawful intervention: 


A prohibited intervention must ... be one bearing on matters in which each 
State is permitted, by the principle of State sovereignty, to decide freely. One of 
these is the choice of a political, economic, social and cultural system, 
and the formulation of foreign policy. Intervention is wrongful when 
it uses methods of coercion in regard to such choices, which must remain free 
ones. The element of coercion, which defines, and indeed forms the very 
essence of, prohibited intervention is particularly obvious in the case of 
an intervention which uses force, either in the form of military action, or 
in the indirect form of support for subversive or terrorist armed activities 
within another State.” 


First, the activity must bear ‘on matters in which each State is permitted, by 
the principle of State sovereignty to decide freely’ (1.e., the domaine résérve, or 
areas where states are free from international obligations and regulation“), 
and second, it must involve ‘methods of coercion in regard to such choices’.*! 
In other words, to constitute a prohibited intervention, the coercive conduct 
in question must impinge upon ‘matters in which a state 1s permitted to de- 
cide freely’, in what is known as its domaine reserve. The principle of non- 
intervention as a rule of customary international law and the required elements 
to establish a violation thereof derives directly from the rights and duties that 
comprise the principle of sovereignty, namely in relation to the protection of 
a state’s political independence. As identified by Ziegler, ‘[n]on-interference 
in the domaine réservé is a fundamental right of States derived from sovereignty 
and protected by the principle of non-intervention in their internal affairs.’ 
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Scope of domaine reserve 


A state’s domaine reserve is a state’s internal affairs, or a sphere of activity which “[is] 
not, in principle, regulated by international law’.** Largely, as a result of globali- 
sation bringing about an increase in interdependence, the further development 
of international law, and an increase in international integration, the scope of 
a state’s domaine réservé is considered to be increasingly limited as fewer areas 
remain free from international rules.” Although traditionally a state’s domaine 
reserve may be considered to encompass the organisation of government, the 
treatment of citizens and their use of territory, today it is not always possible to 
identify entire policy areas that remain within the scope of a state’s domaine réservé 
as many aspects of these matters have been significantly internationalised.°° 

Notwithstanding, in Nicaragua the IC] identified the “choice of a political 
system’ as a clear-cut example of an area within a state’s domaine réservé?” As 
such, cyber activities conducted by foreign states that affect either the process 
by which elections are conducted or their outcome may qualify as a prohib- 
ited intervention, provided the coercion element is also satisfied.>® Given that 
electoral interference falls within a state’s domaine réservé, any conclusions of 
this chapter related to electoral interference are dependent upon the fulfil- 
ment of the domaine résérve element. 


“Interference” and “intervention” 


>> cc: 


The terminology of “intervention”, “interference” and “coercion” are vague 
and often used in political rhetoric, with intervention and interference some- 
times used interchangeably,*’ though “interference” suggests a wider scope 
of activities, especially when used alongside “intervention”.*” Oppenheim 
addresses the terminology of “interference” in consideration that mere inter- 
ference does not amount to a prohibited intervention: ‘Interference pure and 
simple is not intervention’.*! 

This chapter therefore uses the term “interference” to describe a wide 
range of activities that are not necessarily coercive and therefore may not 
amount to a prohibited intervention. “Intervention” is used to describe ac- 
tivities that involve a coercive intervention in the internal or external affairs 
of another state. 


The element of coercion and its application in the 
cyber context 


The coercion element delimits the principle from all lawful forms of state-to- 
state interaction that may have an effect on another state,” playing a crucial 
role in the purpose of the prohibition on intervention, which is ‘to provide an 
acceptable balance between the sovereign equality and independence of states 
on the one hand and the reality of an interdependent world and the inter- 
national law commitment to human dignity on the other'.* In other words, 
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its requirement ‘regulates the line’ between mere interference, which is not 
considered to constitute a violation of international law, and a prohibited in- 
tervention.*" In this sense the coercion element ‘defines, and indeed forms the 


very essence of, prohibited intervention’.” 


Definition of coercion 


Despite its critical role in defining a prohibited intervention, there is no pre- 
cise definition for what constitutes ‘coercion’ in international law.*° Parallels 
with the use of the term in domestic law and philosophical contexts must be 
regarded cautiously,” as the use of the term in international law has its own 
distinct meaning. An examination of coercion as it arises in different con- 
texts in international law reveals the term 1s used inconsistently in its employ- 
ment by various entities.* Article 52 of the Vienna Convention on the Law 
of Treaties deals with coercion in the context of the conclusion of a treaty,” 
safeguarding against the conclusion of a treaty as a result of threat or use of 
force. However, the term coercion within Article 52 is limited to coercion 
by the threat or use of force that must be unlawful to invalidate the treaty in 
question.”! Similarly, Article 18 of the International Law Commission’s Arti- 
cles on State Responsibility offers little assistance in defining coercion, where 
Article 18 involves the allocation of the responsibility that attaches to the act 
of the coerced state to the coercing state (if the act in question constitutes an 
internationally wrongful act).52 The commentary to Article 18 makes it clear 
that the reference to coercion does not necessarily mean ‘unlawful’ coer- 
cion.” However, the commentary provides sparse guidance on the definition 
of coercion beyond stating that most instances within the scope of Article 18 
will be unlawful “because they involve a threat or use of force contrary to 
the Charter of the United Nations, or because they involve intervention, ie 
coercive interference, in the affairs of another State’.°4 

At the same time, it is generally understood that only acts of a certain 
magnitude are likely to qualify as coercive.” In the Nicaragua case, the Court 
drew significant support from the 1970 Declaration on the Principles of 
International Law concerning Friendly Relations and Cooperation among 
States in which the principle of non-intervention features prominently.°° The 
text of the Declaration refers to coercion as follows: 


No State or group of States has the right to intervene, directly or in- 
directly, for any reason whatever, in the internal or external affairs of 
another State. Consequently, armed intervention and all other forms 
of interference or attempted threats against the personality of the State or 
against its political, economic and cultural elements, are in violation of 
international law. No State may use or encourage the use of economic, 
political or any other type of measures to coerce another State in order 
to obtain from it the subordination of the exercise of its sovereign rights 
and to secure from it advantages of any kind.” 
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According to Oppenheim, ‘the interference must be forcible or dictatorial, 
or otherwise coercive, in effect depriving the state intervened against of 
control over the matter in question’.°® For Jamnejad and Wood, ‘[o]nly 
acts ... that are intended to force a policy change in the target state will 
contravene the principle’,°” though “[i]f the target state wishes to impress 
the intervening state and complies freely, or the pressure is such that it 
could reasonably be resisted, the sovereign will of the target state has not 
been subordinated. 6° 

The challenges posed by cyber operations have brought discussion about 
the content of the element of coercion to the forefront of debate among 
states and commentators.°! In the literature on cyber operations, the non- 
intervention principle is often discussed as requiring a relatively high thresh- 
old of violation because of the requirement for the behaviour on the part of 
the perpetrating state to be coercive in nature.’ Drawing on the text of the 
Declaration on Friendly Relations,” the Tallinn Manual 2.0 group of experts 
consider coercion ‘refers to an affirmative act designed to deprive another 
State of its freedom of choice, that is, to force that State to act in an involun- 
tary manner or involuntarily refrain from acting in a particular way"? where 
“coercion must be distinguished from persuasion, criticism, public diplomacy, 
propaganda, retribution, mere maliciousness, and the like in the sense that, 
unlike coercion, such activities merely involve either influencing (as distinct 
from factually compelling) the voluntary actions of the target State or seek no 
action on the part of the target State at all’.°° 

Schmitt, the Director of the Tallinn Manual projects, paraphrases the text of 
the Tallinn Manual 2.0 with the narrower definition that: 


. a coercive act is one designed to compel another state to take action 
it would otherwise not take, or to refrain from taking action it would 
otherwise engage in.°° [Coercion] is accordingly more than mere influence. 
It involves undertaking measures that deprive the target State of choice... 
diplomacy and propaganda, albeit intended to cause another State to act in 
a certain manner, do not qualify as intervention because the target State retains 
the ability to choose; the decisions they are meant to affect remain voluntary, even 
though they may now be suboptimal.” 


Schmitt later elaborates on this position but without reference to the Tallinn 
Manual 2.0: 


Restated, an act of coercion is one that deprives another State of choice 
by either causing that State to behave in a way it otherwise would not 
or to refrain from acting in a manner in which it otherwise would act. 
Merely influencing the other State’s choice does not suffice; the choice to act or 
not has to effectively be taken off the table in the sense that a reasonable State 
in the same or similar circumstances would no longer consider it to be a viable 
option.°8 
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As this threshold of coercion is ‘seldom reached’, the editors of the Tallinn 
Manual 2.0 argue that the vast majority of hostile cyber operations attribut- 
able to states would only implicate a possible violation of sovereignty.” As 
such, they propose that a rule of sovereignty be developed that acts as a “nor- 
mative firewall” to prohibit certain cyber operations below this threshold”” 
that is capable of covering instances of interference with ‘inherently govern- 
mental functions’ even in scenarios where there is no coercion.’! Schmitt and 
Biller argue that the demanding element of coercion required for a prohibited 
intervention means that states who fail to endorse a “rule” of sovereignty 
as Schmitt proposes deprives states of the ‘most likely legal basis’ for tak- 
ing counter-measures and ‘affords other [states the flexibility to act in an 
‘indiscriminate and reckless’ manner while claiming to operate within the 
boundaries of international law’.’* However, it is unclear why if the “rule” 
of sovereignty Schmitt proposes exists whereby an interference with or usur- 
pation of ‘inherently governmental functions’ constitutes a violation of that 
“rule”,’> the ICJ and states made such an effort to identify and develop the 
requirement that an act be coercive to constitute a prohibited intervention. 
While the Tallinn Manual 2.0 group of experts were unable to definitively 
define the term ‘inherently governmental functions’,” the text states that ‘[u] 
surpation of an inherently governmental function differs from intervention 
in that the former deals with inherently governmental functions, whereas the 
latter involves the domaine réservé, concepts that overlap to a degree but that 
are not identical.’’° No further elaboration on this distinction is provided. 
However, it is possible to identify support for a broader understanding of 
coercion in language suggesting various means and techniques whereby a 
state may coerce another state in relation to the exercise of the latter’s state 
power that do not necessarily require the intervention to be limited to “forc- 
ing” a policy or government change,” or compelling ‘another state to take 
action it would otherwise not take, or to refrain from taking action it would 
otherwise engage in’ per se.” The Friendly Relations Declaration refers to 
‘[securing] ... advantages of any kind’, and this formula is reiterated in a num- 
ber of UN General Assembly resolutions and in the Charter of the Organiza- 
tion of American States,’® though the phrase has been criticised for being an 
overstatement that is misleading in what may constitute coercive behaviour.’” 
Oppenheim’s phrasing of ‘the interference must be forcible or dictatorial, or 
otherwise coercive’ implies that coercion may take other forms.5 In the Nicara- 
gua case the IC] determined that coercive behaviour can be direct as well as 
indirect,5! where intervention is wrongful ‘when it uses methods of coercion 
in regard to such choices ... which must remain free ones’ and provided a series of 
different examples that may or may not qualify as an unlawful intervention.’ 
While acts involving forcible pressure may be a more clear case of coercion,85 
publicists who have sought to clarify the content of coercion often envision 
a spectrum of action.**+ As noted by Damrosch, ‘[t]he traditional formulation 
of intervention as “dictatorial interference” resulting in the “subordination of 
the will” of one sovereign to another is... unsatisfactory, because some subtle 
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techniques of political influence may be as effective as cruder forms of dom- 
ination’.®° Kunig considers that intervention ‘aims to impose certain conduct 
of consequence on a sovereign state’, °° Higgins warns against the oversim- 
plification of linear notions based on the invasiveness of acts in determining 
the coercive nature of an act, because not all maximally invasive acts are 
unlawful and not all minimally invasive acts are law ful.5 Joyner considers 
coercion merely ‘involves ... compelling the government of another State to 
think or act in a certain way by applying various kinds of pressure, threats, 
intimidation or the use of force.’ 

Upon closer examination, coercion is capable of a broader definition that 
may be understood as behaviour aimed at seeking an advantage of some kind 
by depriving the target state of its free will over the exercise of its sover- 
eign powers. Moynihan argues that coercion may only require a degree of 
pressure to deprive the target state of control of its state functions,®*” where 
‘coercive behaviour may extend beyond forcing a change of policy to other 
aims, such as preventing the target state from implementing a policy or re- 
straining its ability to exercise its state powers in some way’. While the 
Tallinn Manual 2.0 commentary examples of non-intervention involve clear 
instances of a particular policy decision being forced on the target state,”! 
its definition of coercion appears to support an understanding that ‘could 
include restraining a state from exercising its state functions more broadly, 
as well as forcing it to act in a particular way’:’” ‘[Coercion] refers to an af- 
firmative act designed to deprive another State of its freedom of choice, that is, to 
force that State to act in an involuntary manner or involuntarily refrain from acting 
in a particular way’? For some of the Tallinn Manual 2.0 group of experts, ‘to 
be coercive it is enough that an act has the effect of depriving the State of 
control over the matter in question’.”* This suggests coercion may be satisfied 
in a wider range of scenarios such as operations that aim to merely disrupt 
or undermine the ability of another state to exercise control over its sover- 
eign functions.” In Schmitt's restatement of the definition of coercion in the 
Tallinn Manual 2.0, the quote contains a broader reference to ‘measures that 
deprive the target state of choice’.”° ‘[M]easures that deprive the target state 
of choice’ is far broader than an act ‘designed to compel another state to take 
action it would otherwise not take, or to refrain from taking action it would 
otherwise engage in’, or where ‘the choice to act or not has to effectively be 
taken off the table in the sense that a reasonable State in the same or similar 
circumstances would no longer consider it to be a viable option” For Watts, 
“actions merely restricting a state’s choice with respect to a course of action 
or compelling a course of action may be sufficient to amount to violations 
of the principle of non-intervention’.”* Buchan simply states that coercion 
‘subordinates the will of the state in order for the entity exercising coercion 
to realise certain objectives’.”” 

Alternatively, several authors have argued that the principle of non- 
intervention, usually the coercion element itself, should be circumvented or 
weakened to have a cyber-specific application. Despite these proposals, it 
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is unclear why the same criteria to assess coercion should not apply equally in 
both the cyber and non-cyber context,!”! unless state practice and opinio juris 
results in the development of customary international law that determines 
otherwise. As authors making such assertions provide no state practice or opi- 
nio juris in support of their arguments they constitute lex ferenda. Others have 
suggested that in certain cyber operations that involve electoral interference 
the principle of self-determination may be more relevant in establishing an 
internationally wrongful act, avoiding the need to satisfy the demanding co- 
ercion element required for a prohibited intervention. °? 


Positions of states in the cyber context 


Among the limited number of states that have made statements on how 
they consider international law to apply to cyber operations, several directly 
address the definition of coercion. Positions range from recognising a de- 
manding element of coercion to more broad interpretations. Norway adopts 
a position close to that of Schmitt’s narrower restatement of the text of the 
Tallinn Manual 2.0, whereby coercion ‘compells] the target State to take a 
course of action, whether by act or omission, in a way that it would not 
otherwise voluntarily have pursued’, where ‘cyber activities that are merely 
influential or persuasive will not qualify as illegal intervention’. 1’ Similarly, 
for the Netherlands, ‘[i]n essence [coercion] means compelling a state to take 
a course of action (whether an act or an omission) that it would not otherwise 
voluntarily pursue.’ 104 Romania takes a similar position, whereby coercion 
‘[means] meaning that the goal of the intervention must be to effectively 
change the behaviour of the target State'.1%5 Australia recognises a broader 
understanding of coercion, where ‘[c]oercive means are those that effectively 
deprive the State of the ability to control, decide upon or govern matters of 
an inherently sovereign nature’, 196 Germany adopts a position that is broader 
still, where ‘[c]oercion implies that a State’s internal processes regarding as- 
pects pertaining to its domaine réservé are significantly influenced or thwarted 
and that its will is manifestly bent by the foreign State’s conduct’. New 
Zealand also adopts a broad approach, whereby ‘[c]oercion can be direct or 
indirect and may range from dictatorial threats to more subtle means of con- 
trol’.!° For Switzerland, ‘[t]he distinction between exerting influence, which 
is permissible, and coercion, which is not, must be determined on a case-by- 
case basis’. 0? A number of states including the UK and Australia also appear 
to embrace a broader understanding of coercion given the examples they 
provide of cyber scenarios that may constitute a violation of the principle of 
non-intervention.!!° 

Several of these states recognise that the definition of coercion is yet to 
crystallise, for example, Germany recognises that the coercion element ‘re- 
quires further clarification in the cyber context’!!! while the Netherlands 
notes generally that ‘[t]he precise definition of coercion, and thus of unau- 


thorised intervention, has not yet fully crystallised in international law, 112 
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Intent and outcome 


The covert nature of cyber operations raises questions about the extent to 
which a state’s intent or motivations, and the outcome of the activity in ques- 
tion, may be relevant in assessing whether an act is coercive. 

Literature discussing coercion in the context of cyber operations recog- 
nises that states will usually have a goal in carrying out the action in relation 
to the exercise of its sovereign functions.!? However, while coercive acts 
will by nature involve an intention to compel an outcome or conduct, the 
intention of the intervening state does not necessarily need to match that of 
the party receiving assistance,''* and the motive of the intervening state is not 
considered to be an important factor in determining a violation of the non- 
intervention principle.'!® Nonetheless, the coercive behaviour on the part of 
the perpetrating state will be intentional by nature, and intent is therefore a 
further constitutive element required for a violation of the non-intervention 
principle.'!© 

It is generally understood that whether or not a coercive cyber operation 
produces the desired outcome has no bearing on whether there has been a vi- 
olation of the non-intervention principle," though the effects of the act are 
relevant because of the ‘close causal link between the coercive behaviour and 
its actual or potential effects on the target state’s free will to exercise control 
over its sovereign functions. 115 

Some authors have argued that due to the unique nature of cyber oper- 
ations greater weight should be placed on the actual or potential effects of 
the coercive activity on the target state’s inherently sovereignty functions." "° 
The majority of the Tallinn Manual 2.0 group of experts considered that it is 
not required that the target state has knowledge of the coercive act.!7 For 
example, ifa cyber operation caused an effect that disabled a target state’s in- 
frastructure in a way that caused a “degree of pressure” depriving the target 
state of control of its state functions, but that target state was unaware a cyber 
operation had caused the malfunction, perhaps believing it to be a system 
failure, knowledge of the coercive act is not necessary for there to be a pro- 
hibited intervention. 


Electoral interference and challenges presented by 
artificial intelligence 


In the cyber context, the non-intervention principle is frequently discussed 
in relation to political interference of a state in the affairs of another in rela- 
tion to internal electoral processes. A number of states have directly referred 
to electoral interference in relation to the principle of non-intervention in 
the context of cyber operations. !?! As with political interference generally 
in the non-cyber context,!?” the conduct in question may encompass acts of 
greatly differing intensity and coerciveness. Perhaps the most coercive form 
of political interference is regime change, which has been recognised as a 
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clear violation of the principle of non-intervention.'*° Outside interventions 
involving the use of force, regime change may be achieved through the pro- 
vision of support and funding for insurrectionary opposition groups, as was 
examined in the Nicaragua case which primarily concerned the US funding 
and supporting a political opposition to a foreign government. !?* 

The support of a non-insurrectional political party in a foreign electoral 
process is understood to be less clear in application, although the element 
of coercion remains the ‘key test in determining whether there has been 
a breach of the non-intervention principle.” In the cyber context, online 
disinformation operations often provide support for candidates which are le- 
gitimately within the official electoral process. For Jamnejad and Wood, in 
the context of political interference, the provision of support to a party on the 
eve of an election is a more intrusive act because it is more likely to result in 
a change in government.!~° State practice of electoral interference is exten- 
sive, perhaps because political support leading up to an election may be most 
effective in influencing its outcome. 127 At the same time, numerous General 
Assembly Resolutions condemning interference in electoral processes sug- 
gests that states are particularly concerned about this form of intervention. 1a 


Distinction between direct and indirect interference 


For the following discussion it is helpful to draw a distinction between direct 
interference with election infrastructure, such as a cyber operation targeting 
voting systems to disrupt or alter the results ofan election, and indirect inter- 
ference in influencing or manipulating voter behaviour, which may consist 
of disinformation campaigns on online platforms. 1? In Nicaragua the Court 
found that the principle of non-intervention ‘forbids all States or groups of 
States to intervene directly or indirectly in internal or external affairs of other 
States’.'°° The Court determined that the US funding of the contras consti- 
tuted a prohibited intervention despite the fact that a number of steps took 
place after the US transfer of funds and before the coercive act of the Sandini- 
sta regime of Nicaragua occurred, demonstrating that coercive behaviour can 
be direct as well as indirect. In that sense, the Court ‘adopted a more nuanced 
approach to understanding both coercive behaviour and intervention than 
simply direct, dictatorial behaviour. "°! 

Interference that involves the direct targeting of election infrastructure 
may, for instance, involve a cyber operation that alters election results, or that 
alters the status of voters or removes their eligibility to vote, or that renders 
all votes uncountable or systems unavailable to perform the voting process. 9? 
Examples might include a Distributed Denial of Service attack that renders 
the systems required to count or process votes temporarily unavailable, re- 
sulting in a situation where it is not possible for votes to be cast. As the perpe- 
trating state is attempting to alter the results of the election in a direct manner 
to compel the target state to achieve an outcome, this is likely to amount 
to coercion and constitute a prohibited intervention.!%% A number of states, 
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including Australia, "4 Germany,” Israel," 36 New Zealand,” Norway, "°S 


Romania, °? the UK and the us have identified the direct targeting 
of infrastructure to disrupt or alter data on those systems as a likely violation 
of the non-intervention principle. As Brian Egan noted while serving as US 
Department of State Legal Adviser, ‘a cyber operation by a State that inter- 
feres with another country’s ability to hold an election or that manipulates 
another country’s election results would be a clear violation of the rule of 
non-intervention,’ !” 

While direct interference with election infrastructure is clearly capable of 
being coercive in nature, and so may constitute a prohibited intervention, a 
more complicated issue is whether indirect interference that seeks to influ- 
ence or manipulate voter behaviour may be coercive, and thus constitute a 
prohibited intervention. In discussing the limits of the element of coercion 
and the principle of non-intervention, some scholars have briefly referred to 
advanced artificial intelligence techniques such as deepfakes as an example of 
a particularly problematic method of interference.'4? The development and 
use of technology such as advanced artificial intelligence techniques do not 
alter the criteria by which coercion is examined. However, the challenge 
posed by such technology is whether its use in disinformation and micro- 
targeting, if carried out extensively by the perpetrating state, could through 
influence alone amount to coercion in undermining the target state’s ability 
to conduct a free and fair election to decide its government.'*4 Drawing on 
technical reports on the use of advanced artificial techniques in relation to 
electoral interference it is possible to identify several ways in which such 
techniques may play a significant role in electoral interference, enhancing the 
effectiveness of spreading disinformation on online platforms. '4> 


Influencing an electorate and the use of artificial intelligence 
techniques 


Indirect interference operations that seek to influence voter behaviour may 
involve spreading disinformation on social media platforms where false or 
misleading articles are circulated by actors using fake bot accounts. Artificial 
intelligence technology allows actors to create numerous accounts, using ma- 
chine learning to build fake user profiles and profile photos from user data, 
which can then be controlled as bots en masse to give the impression that 
many citizens of a state share a particular view or opinion." This can be 
used to create a “herd mentality” on online platforms during an election cy- 
cle, enabling foreign actors to have a far more pervasive effect in influencing 
voter behaviour than if these profiles and their activity had to be created and 
maintained manually. The use of advanced artificial intelligence techniques 
such as deepfakes, that is, the creation of synthetic videos that closely resemble 
real videos, #8 allow foreign actors to release and spread videos containing false 
statements or actions of politicians, which may be used to incite political action 
or discredit an individual or political party. One of the reasons deepfakes are so 
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effective in influencing public opinion is their ability to harness confirmation 
bias.1% There are already several examples of this in practice. For example, in 
2019 in Malaysia, a video purporting to show the Minister for Economic Af- 
fairs engaging in a sexual encounter caused significant reputational damage.” 
In 2020, the Oxford Internet Institute identified 81 states that use online plat- 
forms to spread computational propaganda and disinformation, '°' demonstrat- 
ing the extensive nature of state practice in this area. 

According to the position of some who maintain a narrow or demanding 
understanding of the coercion element, influencing voter behaviour through 
disinformation campaigns may not constitute a coercive act because although 
such activity may be ‘a noxious form of influence’, ‘voters (the state) retain 
their ability to decide for whom to vote’,!>? According to such views, merely 
influencing a state’s choice is not coercive as it does not force that state to act 
in an involuntary manner or involuntarily refrain from acting in a particular 
way.I5 3 In relation to these views, for Switzerland, ‘[t]he distinction between 
exerting influence, which is permissible, and coercion, which is not, must be 
determined on a case-by-case basis’,!°4 and for Norway ‘cyber activities that 
are merely influential or persuasive will not qualify as illegal intervention.’ !°° 

However, the fact that an operation seeks to compel an outcome through 
influence alone does not necessarily preclude it from compelling a state to 
act in an involuntary manner or to involuntarily refrain from acting in a 
particular way if that state is deprived of control over its ability to hold a free 
and fair election. The use of advanced artificial intelligence techniques to 
create an impression of a narrative among an electorate on online platforms 
that discredits a candidate or political party pushes the boundaries of the 
interpretations of coercion as outlined in this chapter. Globalisation and ad- 
vances in technology constitute significant changes since the period of major 
UN activity on the principle of non-intervention and these developments 
and challenges will leave their mark on the application of the principle.I5€ 
The effects that are possible to achieve through the use of technology such as 
advanced artificial intelligence techniques are unprecedented in their speed, 
effectiveness and scale in manipulating the behaviour of an electorate to an 
extent that simply wasn’t possible before.” This chapter has demonstrated 
that coercion is capable of a broader application that relies on a degree of pres- 
sure to deprive the target state of control of its state functions, under which 
certain influence operations may amount to coercion. Accordingly, as stated 
by Tsagourias, ‘the use of deep fakes when, during an electoral campaign, 
imageries, voices, or videos of politicians are simulated in order to discredit 
them ... [t]o the extent that such operations are designed and executed in 
such a way as to manipulate the cognitive process where authority and will 
are formed and to take control over peoples’ choices of government ... would 
constitute intervention.’ "8 

The question is therefore not whether influence operations may be coer- 
cive, but instead what factors or conditions are relevant in determining in 
which scenarios they may amount to coercion. The provision of information 


Advanced artificial intelligence techniques and the principle of non-intervention 237 


on the eve of an election is a more intrusive act and the falseness of infor- 
mation may indicate a higher likelihood that this activity may interfere with 
a democratic state’s inherent right to hold free and fair elections.'°? In con- 
sideration of whether propaganda may be coercive, Murty asks whether the 
audience’s choice of alternatives have been severely restricted as a result of the 
operation, |©° which implies a focus on outcome, actual or expected, rather 
than means or methods employed.'*! For Watts ‘[o]nly where propaganda 
could be said to constitute significant support for coercion on a level com- 
mensurate with the logistical and financial support recognized by the Nica- 
ragua Court, could cyber propaganda amount to a violation of the principle 
of non-intervention.’!©” For Jamnejad and Wood, whether the dissemination 
of propaganda violates the non-intervention principle depends on all the cir- 
cumstances at hand: If the information that is shared is factual and neutral, 
then it is not likely to constitute a breach of the non-intervention principle.165 
Conversely, if the information is disinformation, that is, false or not factually 
accurate, such as the discussed uses of advanced artificial intelligence tech- 
niques, the likelihood that this activity may interfere with a democratic state’s 
inherent right to hold free and fair elections is higher. 


Influence operations that result in disruption of election process 


It is also possible that the spreading of disinformation that may include the use 
of advanced artificial techniques in influence operations could, though not 
directly targeting the infrastructure of an election, nonetheless disrupt the 
election process in a way that would significantly alter the outcome or results 
of an election, tantamount to a direct operation that targets and disrupts or 
alters election infrastructure systems, !©° Imagine a scenario where, on the 
eve of an election, an incumbent head of state appeared to announce that the 
election was cancelled or postponed, or that the individual was conceding 
or withdrawing from the race, the deepfake video of which spreads virally 
online resulting in a significant number of voters not casting their vote on 
the day of the election. While not a direct attack on voting systems to alter 
or disrupt those systems, such a scenario may still produce the same effect. 
In this case, as the perpetrating state is attempting to alter the results of the 
election in a manner that deprives the target state of control over its ability to 
conduct a free and fair election to decide its government, equivalent to the 
direct electoral interference highlighted by several states as an example of a 
prohibited intervention.'©° Such a scenario would likely to amount to coer- 
cion and constitute a prohibited intervention. !°” 


Individual influence operations as part of a broader influence 
campaign 


While foreign interference in the 2016 US elections has received signifi- 
cant attention from scholars of international law on questions relating to the 
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principle of non-intervention,°* it nonetheless remains a particularly useful 


case study to demonstrate several issues relating to indirect electoral inter- 
ference that concern influencing or manipulating voter behaviour within 
the context of a larger influence campaign. The influence operation had 
two main components: The exfiltration and publication of the Democratic 
National Committee materials, and the spreading of disinformation on social 
media to influence voter choice.'©? 

Regarding the first component of the exfiltration and publishing of mate- 
rial, the material was verified as authentic,” and many subsequent news arti- 
cles based on its publication were both factual and in the public interest. That 
this step may constitute a prohibited intervention is not obvious,'”! because 
the published material is factual it is less likely to be coercive and constitute a 
breach of the non-intervention principle.'’* However, if a scenario took place 
whereby sensitive state secrets or classified documents were exfiltrated by 
state actors who threatened to publish or disclose said documents unless the 
state changed their policy on a particular matter within its domaine résérve, that 
action may be coercive and thus would constitute a prohibited intervention. 

For the second element of spreading falsified news on online platforms 
to influence the election results, the information was false and not factually 
accurate and was disseminated on the eve of an election,!”* so the likelihood 
that this activity may be coercive in interfering with a democratic state’s 
inherent right to hold free and fair elections is higher!” and depending on 
these factors and the definition of coercion outlined above, the operation may 
constitute a prohibited intervention.” 

Some authors simply state that it is unclear whether the operations were 
coercive and therefore that they did not constitute a prohibited interven- 
tion,” while others maintain it violated the principle based on a broad or 
modified interpretation of coercion.'”’ In relation to those that do not con- 
sider the operations to be coercive, Schmitt recognises what he considers 
to be “a slightly sounder view’, that ‘the cyber operations manipulated the 
process of elections and therefore caused them to unfold in a way that they 
otherwise would not have’, and so may be coercive.!’8 This position appears 
to contradict his assertion in the same paper that ‘^... diplomacy and propa- 
ganda, albeit intended to cause another State to act in a certain manner, do 
not qualify as intervention because the target State retains the ability to choose; the 
decisions they are meant to affect remain voluntary, even though they may now be 
suboptimal’.!’? The operations constituted indirect interference as they did 
not affect the state or the electorate’s ability to choose a candidate during the 
election in so far as the decisions of the electorate remained voluntary. This 
debate demonstrates the difficulty in maintaining the position that influence 
operations alone are not capable of amounting to coercion, which will only 
be exacerbated by the increased use and effectiveness of advanced artificial 
intelligence techniques in future influence operations. 

While in practice it is difficult to separate individual influence operations 
from larger campaigns of influence, Schmitt has suggested that the scale and 
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effects test from the ICJ’s Paramilitary Activities judgment assessing whether 
a use of force rises to threshold of an armed attack might be adopted in the 
context of intervention, and cites the recent position of Germany!*? which 
he suggests supports this proposal.'*! However, despite Schmitt’s claim that 
‘(t]he German adoption of the approach in the context of intervention is fur- 
ther evidence that it is gaining widespread acceptance as a means of assessing 
international law thresholds more generally when applied to cyber operations’, 
which he considers to be “an appropriate use of the scale and effect stand- 
ard’,'®? there is sparse evidence in literature and among states supporting the 
assertion that this particular standard that the ICJ formulated in the context 
of use of force is directly transposable to the principle of non-intervention. 
Rather, the statement by Germany is better understood as a general statement 
recognising that individual cyber operations often form part of a larger cam- 
paign of influence that may be coercive as a whole and thus may constitute 
a prohibited intervention. It remains to be seen whether more states will 
recognise such an approach in light of the further development and increased 
availability of technology such as advanced artificial intelligence techniques 
and their ability to enhance the effectiveness of disinformation campaigns. 15? 


Conclusion 


The element of coercion plays a key role that ‘regulates the line’ between 
mere interference, which is not generally considered to constitute a violation 
of international law, and a prohibited intervention.'** The coercive behav- 
iour on the part of the perpetrating state will be intentional by nature, and 
intent is a further constitutive element required to establish a prohibited in- 
tervention. Though the ICJ has recognised the principle of non-intervention 
is a rule which has been frequently breached, it confirmed this has not af- 
fected its status as a rule of custom, and that it remains ‘part and parcel of 
customary international law.155 

Some commentators in the literature on the application of international 
law to cyber operations have asserted that the element of coercion sets a de- 
manding standard that is seldom met. However, this chapter argues that the 
non-intervention principle is capable of a broader application where coercion 
requires the exertion of a “degree of pressure” to deprive the target state of 
control of its state functions, where coercive behaviour may extend beyond 
forcing a change of policy to preventing the target state from implementing a 
policy or restraining its ability to exercise state powers. Indeed, several states 
have recently recognised broader and more nuanced understandings of coer- 
cion under which influence operations may qualify as prohibited interven- 
tions. However, it is important to note that to date no state has explicitly 
claimed a cyber operation has violated the principle of non-intervention. 

The primary focus of the application of the rule of non-intervention to 
cyber operations among states and commentators concerns electoral inter- 
ference. While several states now recognise the application of the principle 
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of non-intervention to scenarios of electoral interference that directly target 
election infrastructure to disrupt or alter the result of an election, !*” a more 
complicated issue is whether indirect operations that seek to influence or 
manipulate voter behaviour may be coercive, and thus constitute a prohibited 
intervention. 

Advances in technology increasingly enable indirect influence operations 
to be capable of amounting to coercion insofar as the have the ability to un- 
dermine the target state’s ability to conduct a free and fair election to decide 
its government, in effect applying a “degree of pressure” to deprive the target 
state of control of its state functions. Developments in technology such as the 
use of advanced artificial intelligence techniques in influence operations do not 
change or alter the criteria on which coercion is evaluated. However, they am- 
plify the scale of effects that are possible to achieve from influence operations 
which increases the likelihood that such activity could interfere with a demo- 
cratic state’s inherent right to run free and fair elections. As such, they increas- 
ingly push the boundaries of indirect interference operations that influence or 
manipulate voter behaviour, and methods utilising such techniques are capable 
of being coercive and may therefore constitute a prohibited intervention. 

In that sense, the continued development of technology such as advanced 
artificial intelligence techniques that enhance the effectiveness of disinfor- 
mation operations may be said to constitute a challenge to the traditional 
understanding that mere influence alone is not coercive and therefore may 
not qualify as a prohibited intervention. Accordingly, the question is there- 
fore not whether influence operations may be coercive, but instead what 
factors or criteria are relevant to consider in determining in which scenarios 
they may amount to coercion. Individual cyber operations or disinformation 
operations have the ability to be effective in influencing an electorate as a 
whole, and states have begun to discuss whether individual operations which 
may not themselves be coercive may collectively amount to coercion and 
thus be capable of constituting a prohibited intervention. As in practice it is 
difficult to distinguish between individual influence operations that are part 
of a broader campaign of activity, it seems logical that cumulative effects of 
individual operations must be capable of resulting in a “degree of pressure” to 
deprive the target state of control of its state functions. More clarity is likely 
to emerge as states continue to engage in international fora and release state- 
ments outlining their positions. 
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